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AMES  is  a  potential  tool  for  managers  to  help  identify  those  aspects  of 
maintenance  with  large  payoffs  for  improving  operational  readiness  and/or  mis¬ 
sion  completion  rate.  Additionally,  AMES  can  be  used  to  evaluate  quantitative¬ 
ly  the  effects  of  human  factors  improvements  on  aircraft  readiness. 

AMES  is  important  to  the  maintenance  community  because  of  its  treatment  of 
maintenance  error  rates.  Maintenance  errors  are  known  to  have  a  large  impact 
on  many  other  factors  in  the  maintenance  performance  equation,  e.,g.,  elapsed 
time,  consumption  of  spares,  aborted  missions,  etc.  However,  this  knowledge 
has  seldom  been  put  to  good  use.  One  reason  is  that  errors  are  difficult  to 
measure.  Another  reason  is  that  the  effects  of  errors  are  highly  interactive, 
and  not  easily  traced  by  conventional  analysis  techniques. 

AMES  addresses  both  of  these  problems. 

1.  The  measurement  of  error  is  accomplished  by  a  special  data  analysis 
procedure,  which  employs  engineering  inference  to  extract  error 
incidents  from  3-M  records  and  squadron  operating  records. 

2.  The  interactive  effects  of  errors  are  handled  effectively  by  the  com¬ 
puter,  based  on  prior  analysis  of  functional  relationships  in  the 
operation  and  maintenance  environment. 

Preliminary  AMES  results  show  the  enormous  magnitude  of  maintenance  errors 
The  following  results  were  obtained  for  the  F-14A  in  the  base  years  of  1976  and 
1977, 

o  Eighteen  percent  of  maintenance  actions  resulted  in  errors.  This 
number  is  very  conservative  because  many  errors  cannot  be  detected 
with  the  3-M  data. 

o  A  90  percent  reduction  of  these  errors  increases  mission  completion 
rate  from  49.6  percent  to  57  percent.  This  is  equivalent  to  having  1.8 
percent  additional  aircraft  in  each  squadron  of  12. 

o  When  missions  must  be  flown,  mission  completion  rate  may  be  a  better 
measure  of  system  performance  than  operational  readiness.  As  mainte¬ 
nance  improves,  more  missions  are  flown  which  in  turn  results  in  more 
malfunctions.  The  additional  missions  flown  are  reflected  directly 
in  mission  completion  but  only  indirectly  in  operational  readiness. 

The  90  percent  reduction  in  errors  is  believed  possible  using  state-of- 
the-art  aiding/training  technology.  Study  of  the  efficacy  of  aiding/training 
seems  to  be  an  area  of  research  that  could  lead  to  potentially  very  large  cost 
benefits. 

In  addition  to  human  performance  variables,  AMES  includes  as  inputs, 
operating  schedules,  parts  availability,  equipment  item  realiability  values, 
maintenance  manpower  levels,  and  time-to-repair  values.  All  such  data  are 
directly  available  from  existing  records. 

AMES  has  now  been  taken  through  the  prototype  phase  of  model  development. 
It  has  been  shown  to  perform  as  intended  relative  to  the  F-14A  aircraft. 
Recommendations  for  future  use  include:  systematic  model  improvement,  exten-- 
sion  to  other  weapon  systems,  and  involvement  in  trade-off  studies  indicating 
potentially  cost  effective  human  factors  research. 
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PREFACE 

The  Naval  Training  Equipment  Center  ( NAVTRAEQU I PCEN )  has  a  con¬ 
tinuing  interest  in  the  application  and  evaluation  of  the  methods 
used  in  training  design  and  development.  Intrinsic  to  that 
interest  is  the  need  to  measure  performance,  both  within  the  train¬ 
ing  process  and  within  the  work  environment  supported  by  training. 

The  project  described  here  concerns  performance  measurement  in 
aircraft  maintenance. 

This  project  is  part  of  a  larger  effort  to  establish  a  quantita¬ 
tive  relationship  between  the  effectiveness  of  maintenance  and 
the  operational  readiness  of  Fleet  units  served  by  maintenance 
personnel.  Two  other  projects  deal  with  factors  known  to  in¬ 
fluence  maintenance  effectiveness.  Such  factors  include  quality 
of  training;  availability  and  usability  of  technical  information, 
and  maintainability  of  the  hardware. 

In  the  first,  a  study  performed  in  the  period  1978-1980,  the 
relationship  between  maintenance  effectiveness  and  maintenance 
training  was  explored.  A  connection  was  established  between  the 
performance  problems  of  maintenance  technicians  and  the  methods 
employed  in  the  training  of  such  personnel.  This  study  was  funded 
by  NAVTRAEQUIPCEN  under  Contract  N61339-77-0166 . 

In  a  second  study,  the  relationship  between  maintenance  effective¬ 
ness  and  equipment  design  for  the  maintainer  is  being  explored. 
Using  the  AMES  model  as  a  processing  tool,  an  effort  will  be  made 
to  predict  the  impact  on  system  cost/readiness  where  design-for- 
the-maintainer  remedies  are  postulated.  This  study  is  funded  by 
the  Naval  Air  Development  Center  (NAVAIRDEVCEN)  under  Contract 
N6  226 9-7 9- 62-93093. 

The  work  for  the  AMES  program  was  performed  by  the  Xyzyx  Informa¬ 
tion  Corporation,  under  contract  N61339-77-D-0028 .  Close  coop¬ 
eration  was  rendered  by  the  Maintenance  Support  Office  Department 
in  Mechanicsburg;  the  OPNAV  Instruction  Office  in  Washington,  D.C. 
F-14A  squadron  personnel  at  NAS  Miramar;  and  SH-2F  detachment 
personnel  at  NAS,  North  Island. 

There  were  three  objectives  for  the  project.  One  objective  was 
to  provide  a  basis  for  training  new  maintenance  managers  to  under¬ 
stand  the  maintenance  functions  collectively  as  a  system.  A 
second  objective  was  to  provide  a  tool  enabling  R&D  managers  to 
assign  higher  priorities  to  people-related  efforts  in  maintenance 
research. 

The  third  objective  was  a  research  and  development  objective. 

This  objective  was  to  create  a  prototype  functional  simulation 
model  for  studying  aircraft  maintenance  effectiveness. 
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SECTION  I 

INTRODUCTION 


It  is  reasonably  well  known  that  maintenance  technicians  commit 
errors.  Even  the  general  consequences  of  these  errors  are  fairly  well 
known.  These  consequences  include  damaged  equipment,  wasted 
materials,  accidents  and  injuries,  excessive  use  of  spare  parts,  and 
most  prominently,  the  persistent  need  for  rework. 

Curiously,  however,  the  problem  has  not  been  studied  very 
systematically.  Major  efforts  have  been  made  to  measure  and  control 
performance  time,  but  performance  quality  has  gone  unquestioned.  In 
this  respect,  the  maintenance  research  coirmunity  has  taken  the  same 
path  as  was  taken  earlier  by  the  reliability  community.  Neither  has 
constructed  a  viable  method  of  dealing  with  the  errors  they  know  are 
occurring. 

What  is  needed  is  a  practical  way  of  measuring  both  performance  time 
and  performance  quality,  including  the  interrelations  they  represent. 
These  are  the  elements  that  comprise  the  true  variable,  maintenance 
effectiveness.  Given  those  measurements  of  performance  time  and 
quality,  it  is  necessary  to  be  able  to  relate  them  to  a  higher-level 
variable  of  great  importance,  such  as  operational  readiness.  Finally, 
there  is  a  need  for  an  economical  method  of  exercising  this 
relationship  under  varied  conditions. 

Such  a  method  would  make  it  possible  to  study  alternatives  when 
seeking  ways  to  improve  performance  at  the  operational  level.  The 
effects  of  changes  in  the  personnel  subsystem  could  be  examined 
quantitatively,  thus  putting  them  on  an  equal  footing  with  proposed 
hardware  solutions.  This  capability  would  have  large  implications  for 
the  training  community,  which  has  always  had  difficulty  in 
demonstrating  impact  at  the  bottom  line. 

OBJECTIVES 

Our  immediate  goal  is  to  study  the  relationship  of  maintenance 
effectiveness  to  system  performance.  A  functional  simulation  model  of 
the  aircraft  flight  and  maintenance  process  can  be  used  to  quantify 
this  relationship.  The  immediate  objective  of  this  project  is  then  to 
create  a  prototype  functional  simulation  model  for  studying  aircraft 
maintenance  effectiveness. 
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A  more  distant  goal  is  to  provide  tools  and  techniques  to  improve 
maintenance.  The  long  term  objectives  should  be  chosen  to  use  the 
functional  simulation  model  as  a  first  step.  The  two  long  term 
objectives  are  as  follows: 

•  to  provide  a  basis  for  training  new  maintenance  managers  to 
understand  the  maintenance  function  as  a  system. 

•  to  provide  a  tool  enabling  R&D  managers  to  assign  higher  prioritif s 
to  people-related  efforts  in  maintenance  research. 

ORGANIZATION  OF  THIS  REPORT 

This  report  describes  the  AMES  Model.  The  report  is  organized  in  the 
following  way: 

•  The  first  three  sections  provide  an  introduction  and  overview  of 
the  project.  Section  I  provides  a  brief  overview.  Section  II 
describes  errors  and  their  effects  on  maintenance.  Section  III 
discusses  the  modeling  approach  and  why  it  was  chosen. 

•  The  second  three  sections  describe  various  aspects  of  the  AMES 
Model.  Section  IV  describes  the  procedures  used  to  design,  construct 
and  test  AMES.  Section  V  describes  the  additional  information 
required  to  use  the  model.  Section  VI  describes  the  AMES  Model 
itself. 

•  The  final  two  sections  provide  results,  conclusions  an] 
suggestions.  Section  VII  describes  results  of  our  study  an] 
preliminary  results  obtained  from  the  model.  Section  VIII  presents 
conclusions  and  suggestions  for  improvements,  modifications,  and 
potential  uses  of  the  model. 

•  The  Appendix  describes  the  model  in  much  greater  detail. 
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SECTION  II 

MAINTENANCE  ERRORS 


Maintenance  errors  are  discussed  under  the  following  headings: 

•  Kinds  of  Errors 

•  Effects  of  Errors 

•  Measurement  of  Errors 

The  dynamics  of  errors  tend  to  be  similar  in  nature,  regardless  of  the 
maintenance  environment.  However,  the  discussion  here  applies 
specifically  to  the  area  studied;  that  is,  organizational  maintenance 
on  carrier-based  Navy  aircraft. 

KINDS  OF  ERRORS 

As  shown  in  Table  1,  maintenance  errors  may  be  conveniently  sorted 
into  three  categories:  Type  I,  Type  II,  and  Type  d. 

A  Type  I  error  occurs  vhen  the  technician  troubleshoots  the  equipment 
and  concludes  that  a  particular  unit  has  failed  when,  in  fact,  it  has 
not. 

A  Type  II  error  occurs  when  the  technician  troubleshoots  the  equipment 
and  concludes  that  it  is  okay  vrtien,  in  fact,  it  contains  a  defect. 

A  Type  d  error  occurs  when  the  technician  damages  the  equipment  or 
performs  a  corrective  or  preventive  maintenance  action  improperly. 
Examples  would  be  loose  connectors,  mismatched  parts,  improper 
alignment,  and  inadequate  lubrication. 

The  method  of  categorization  used  here  reflects  the  manner  in  which 
records  are  kept  in  the  Navy  Maintenance  and  Material  Management  (3-M) 
System.  This  is  very  important  to  the  modeling  effort.  Any 
measurement  method  seeking  to  impact  the  Naval  Air  Maintenance 
community  must  be  as  consistent  as  possible  with  the  3-M  System  and 
its  associated  data  base. 
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TABLE  1.  KINDS  OF  MAINTENANCE  ERRORS 


TYPE  OF  TYPICAL 

ERROR  EXPLANATION  OF  ERROR  SOURCE  OF  ERROR 


I 

II 


d 


Technician  replaces  a  unit  that  Troubleshooting 

has  not  malfunctioned. 

Technician  fails  to  recognize  a  Troubleshooting; 

unit  that  has  malfunctioned  or  checkout 

been  improperly  handled. 


Techician  fails  to  accomplish  a 
corrective  or  preventive  action 
properly. 


Removal/installation; 
service;  repair;  ad- 
just /align. 
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A  key  fact  seen  in  Table  1  is  that  each  type  of  error  tends  to  be 
associated  with  its  own  particular  maintenance  function  or  functions. 
This  is  a  crucial  property  in  the  design  of  the  AMES  Model.  The 
functions  themselves  are  identified  as  parts  of  the  maintenance  system 
being  simulated.  The  presence  of  three  kinds  of  technician  errors, 
occurring  at  unique  points  within  that  system,  provides  the  basis  for 
a  logical  network.  Within  that  network,  errors  can  be  expressed 
quantitatively  and  their  side  effects  traced  and  measured. 

EFFECTS  OF  ERRORS 

This  discussion  of  the  effects  of  errors  is  an  overview  to  show  the 
general  behavior  of  errors  in  the  maintenance  environment. 

The  effects  of  the  three  types  of  maintenance  errors  are  summarized  in 
Table  2.  As  shown  there,  errors  may  be  discovered  during  post-job 
inspection,  before  a  flight,  or  during  a  flight.  Following  discovery, 
the  effects  may  be  to  abort  the  flight,  repeat  the  maintenance 
function,  use  spare  parts  wastefully,  place  an  unnecessary  load  on 
intermediate  maintenance,  and,  of  course,  incur  a  risk  of  injury  or 
accident. 

Abort  Flight 

Other  than  the  risk  of  injury  or  accident,  which  is  present  whenever 
an  error  is  coirmitted  in  aviation  maintenance,  the  most  serious  effect 
of  error  is  an  aborted  flight.  All  equipment  items  do  not  share 
equally  in  this  risk.  Flights  are  aborted  only  when  critical 
components  are  involved.  Criticality  is  determined  by  a 
pre-establ ished  list.  Note  that  errors  detected  and  corrected  during 
post- job  inspection  normally  do  not  cause  a  flight  to  be  aborted. 
Note  also  that  the  (abort)  effect  is  identical  for  all  three  types  of 
error. 

Repeat  Function 

This  effect  points  up  the  cardinal  rule  of  maintenance  productivity: 
"Every  error,  regardless  of  the  circumstances,  creates  a  need  for 
rework.  Errors  thus  represent  sheer  dead  weight  in  the  system." 

The  extent  of  the  necessary  rework  varies  with  the  type  of  error.  Fbr 
Type  II  and  Type  d  errors,  rework  is  usually  limited  to  the  offending 
maintenance  function,  i.e.,  checkout  or  repair.  For  Type  I  errors, 
however,  the  rework  is  usually  more  extensive.  Not  only  does 
troubleshooting  have  to  be  repeated,  but  the  associated  corrective 
action  must  be  repeated. 
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TABLE  2.  EFFECTS  OF  MAINTENANCE  ERRORS 


Kind 

Of 

Error 

When  Error 

Is  Discovered 

Effects 

Abort 

Flight 

Repeat 

Function 

Use  Spare 
Wastefully 

Place  Load 

On  IMA 

Risk  Injury/ 
Or  Accident 

Post-Job 

•(B) 

• 

• 

• 

♦*» 

C  _ 

Inspection 

=  1 

TJ  m 

Before 

•  (A) 

•(B) 

• 

• 

• 

Is- 

Flight 

a  o c 

# 

During 

•  (A) 

•  (B) 

• 

• 

• 

Flight 

4-* 

Post-Job 

• 

• 

Z 

Inspection 

? 

c  0) 

=>  s 

Before 

•  (A) 

• 

• 

S  S 

Flight 

CD  DC 

During 

•  (A) 

• 

• 

Flight 

Post— Job 

•  (C) 

•  (D) 

• 

S 

Inspection 

o  > 

Q1 
-  £ 

Before 

•  (A) 

• 

•  (C) 

•  (D) 

• 

a  o 

Flight 

DC  ^ 

During 

•  (A) 

• 

•  (C) 

•  (D) 

• 

■d 

Flight 

(A)  Flights  aborted  by  critical  equipment  items  only. 

(B)  "Function"  includes  troubleshooting  &  repair. 

(C)  Spare  parts  wasted  only  when  broken. 

(D)  Load  placed  on  Intermediate  Maintenance  Activity  (IMA)  only  when  broken  part  repairable. 
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Rework  produces  two  kinds  of  negative  results.  First,  and  most 
obviously,  it  increases  the  direct  cost  of  maintenance  by  requiring 
the  presence  of  a  larger-than-optimum  staff.  Second,  it  impedes  the 
work  schedule,  thus  fostering  the  creation  of  queues,  wherein  jobs  are 
waiting  for  people.  In  this  latter  sense,  excessive  demands  for 
rework  always  impinge  upon  aircraft  availability. 

Use  Spare  Wastefully 

Spare  parts  are  used  wastefully  in  two  of  the  three  types  of  error 
situations,  regardless  of  the  time  of  discovery.  The  parts  problem  is 
avoided  only  in  those  cases  where  the  error  is  failing  to  see  that  a 
replacement  is  necessary. 

The  greatest  waste  of  spare  parts  probably  occurs  in  troubleshooting 
because  of  the  trial-and-error  approach  taken  by  so  many  technicians. 
It  is  true  that  good  units  removed  in  error  may  be  used  again  after 
being  re-cycled  through  Intermediate  Maintenance  Activity  (IMA). 
Nevertheless,  while  they  are  in  that  (re-cycling)  mode,  they  are  not 
available  for  re-issue.  Temporary  shortages  are  thus  generated,  often 
leading  to  reductions  in  aircraft  availability. 

The  waste  may  be  less  in  the  "improper  repair"  category,  because  so 
many  repairs  can  be  corrected  without  the  need  for  new  parts. 
However,  that  situation  is  counterbalanced  by  the  occasional  repair 
error  that  causes  such  serious  damage  that  the  broken  parts  cannot  be 
mended,  even  in  IMA.  In  such  cases,  the  spares  inventory  is  impacted 
permanently. 

Place  Load  on  IMA 

Intermediate  maintenance  shops  are  burdened  unnecessarily  every  time  a 
part  is  replaced  in  error.  This  contribution  to  the  problem  is  the 
Type  I  error  committed  in  troubleshooting.  IMA  must  handle  all  such 
parts  as  though  they  were  defective.  In  fact,  only  after  inspection 
and/or  testing  at  IMA  is  it  safe  for  the  parts  to  be  declared  ready 
for  issue.  Since  such  parts  enter  IMA  without  defect,  all  work  done 
on  them  must  be  regarded  as  wasted  effort.  The  result  is  an  increase 
in  the  direct  cost  of  maintenance. 

The  dynamics  are  slightly  different  when  repair  is  done  incorrectly  at 
the  organizational  level.  In  such  cases,  parts  go  to  IMA  only  if  they 
are  damaged.  The  nature  of  the  waste  lies  in  the  time  spent  by  IMA  in 
mending  the  broken  parts.  This  waste  is  relieved  only  when  the  parts 
are  broken  so  badly  that  they  cannot  be  repaired. 
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Risk  of  Injury  or  Accident 

As  indicated  earlier,  the  risk  of  injury  or  accident  is  present  every 
time  an  error  is  committed  in  aircraft  maintenance.  Depending  on 
severity,  injuries  and  accidents  may  impact  all  aspects  of  maintenance 
and  operational  productivity.  That  is,  they  can  affect  direct  costs, 
aircraft  availability,  personnel  availability,  the  spares  inventory, 
support  equipment  availability,  and  even  facility  availability.  They 
are  also  known  to  be  highly  disruptive  of  normal  operations. 

MEASUREMENT  OF  ERRORS 

The  topic  of  error  measurement  is  treated  in  detail  in  Section  V  of 
this  report.  The  purpose  of  the  present  discussion  is  merely  to  show 
that  error  measurement  is  entirely  feasible,  given  real-world  data  of 
the  quality  provided  by  the  3-M  System. 

All  of  the  error  measurements  prescribed  for  the  AMES  Model  can  be 
obtained  from  the  3-M  data  and  many  can  be  identified  by  nunber  in  the 
3-M  Catalog.  Where  necessary,  additional  procedures  are  provided  by 
Xyzyx  to  guide  the  analyst  in  extracting  error  data  from  those 
records. 

Type  I  Error  —  Good  Unit  Replaced 

Information  on  Type  I  errors  is  obtained  from  a  3-M  report  generated 
in  IMA.  The  report  is  MSOD  4790.A2551-01 ,  "No  Defect  Item  Analysis 
Summary."  This  report  summarizes  equipment  items  processed  by 
Aircraft  Intermediate  Maintenance  Departments  for  which  reported 
defects  could  not  be  duplicated.  Such  items  thus  reflect  erroneous 
removals  from  the  aircraft. 

Type  II  Error  —  Bad  Unit  Not  Replaced 

Information  on  Type  II  errors  is  obtained  by  a  time  sequence  analysis 
of  3-M  records  CT11,  CT21  and  CT41.  The  analysis  is  inferential  in 
nature.  Where  successive  flights  are  accompanied  by  corrective 
maintenance  act  ions  on  the  same  system,  and  those  actions  suddenly 
stop,  a  particular  condition  may  be  inferred.  That  is,  the  final 
corrective  action  was  successful,  but  each  preceding  action  must  have 
been  in  error.  Analysis  for  Type  II  errors  requires  engineering 
knowledge  covering  system  equipment,  operating  theory,  and  maintenance 
practices. 
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Type  d  Error  —  Repair  Done  Incorrectly 

Information  on  Type  d  errors  is  obtained  from  malfunction  codes 
exercised  on  3-M  records  CT11  and  CT21.  Two  subsets  of  the 
Malfunction  Code  list  appear  in  Tables  3  and  4.  The  codes  shown  in 
Table  3  indicate  the  definite  occurrence  of  a  Type  d  error.  The  codes 
shovffi  in  Table  4  indicate  a  probable  Type  d  error. 

Probable  errors  must  be  verified  through  reverse  time  sequence 
analysis  vherein  records  are  checked  for  prior  maintenance  actions. 
If  the  equipment  has  been  worked  on  in  the  two  days  immediately 
preceding  discovery  of  the  problem,  the  defect  is  assumed  to  have  been 
induced  by  maintenance  rather  than  by  equipment  failure.  Engineering 
knowledge  is  needed  to  distinguish  between  equipment  failure  and 
maintenance  error. 
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TABLE  3.  MALFUNCTION  CODES  ON  RECORDS  CT1 1  AND  CT21 
INDICATING  DEFINITE  TYPE  d  ERROR 


CODE 

DESCRIPTION 

086 

Improper  handling 

087 

Improper  identification 

246 

Improper  or  faulty  maintenance 

301 

Foreign  object  damage  (FOD) 

304 

FOD-Self  inducted 

651 

Air  in  system 

931 

Inadvertent  operation 

TABLE  4.  MALFUNCTION  CODES  ON  RECORDS  CT1 1  AND  CT21 
INDICATING  PROBABLE  TYPE  d  ERROR 


CODE 

DESCRIPTION 

? 

093 

Missing  part 

105 

Loose  bolts,  nuts,  screws,  rivets,  fasteners,  etc. 

106 

Missing  bolts,  nuts  or  screws 

108 

Broken,  faulty,  or  missing  safety  wire  or  key 

127 

Adjustment  or  alignment  improper 

135 

Binding,  stuck,  jammed 

410 


Lack  of,  or  improper  lube 
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SECTION  III 

GENERAL  DISCUSSION  OF  MODELS 


INTRODUCTION 

The  goal  of  all  aircraft  maintenance  is  to  provide  aircraft  capable  of 
flying  missions.  Therefore,  the  effectiveness  of  maintenance  should 
be  measured  in  terms  of  system  performance,  i.e.,  how  well  the 
aircraft  system  performs  its  mission.  Figure  1  shows  the  factors  that 
contribute  to  or  affect  such  system  performance.  In  the  figure,  the 
complicated  set  of  interactions  involved  in  aircraft  maintenance  is 
included  in  the  circle  labeled  "AIRCRAFT  FLIGHT  &  MAINTENANCE  SYSTEM". 

The  outputs  of  the  "AIRCRAFT  FLIGHT  &  MAINTENANCE  SYSTEM"  circle  are 
two  system  performance  parameters:  operational  readiness,  and  mission 
completion  rate.  Flight  and  maintenance  are  included  together  since 
both  are  required  to  produce  successful  missions  as  well  as  to 
maintain  a  reasonable  level  of  operational  readiness. 

The  most  obvious  way  to  determine  the  extent  to  which  system 
performance  is  affected  by  change  of  the  factors  is  to  observe  an 
actual  squadron.  However,  the  primary  purpose  of  aircraft  is  to  fly; 
it  is  not  to  study  maintenance.  The  high  cost  of  aircraft  prevents 
the  exclusive  use  of  a  squadron  for  the  study  of  maintenance. 
Therefore,  we  need  some  other  ways  to  determine  how  the  various 
factors  mentioned  above  affect  the  aircraft  flight  and  maintenance 
system.  TVo  important  properties  of  a  maintenance  system  need  to  be 
considered: 

•  Complex  Interactions  Between  Factors 

•  The  Random  Nature  of  Aircraft  Component  Failure 

Complex  Interactions  Between  Factors 

Each  of  the  factors  shown  as  inputs  in  Figure  1  is  related  to  some 
part  of  the  flight  and  maintenance  process.  The  immediate  effects  of 
each  of  the  factors  can  be  expressed  in  a  relatively  simple  manner. 
For  instance,  if  personnel  required  to  prepare  an  aircraft  for  flight 
are  not  available,  the  aircraft  cannot  be  made  ready  to  fly.  However, 
because  of  the  complex  interactions  between  the  factors,  it  is 
extremely  difficult  to  determine  how  they  will  affect  the  overall 
performance  of  the  system,  as  expressed  in  terms  of  operational 
readiness  and/or  successful  mission  completion. 
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Maintenance  has  seldom  been  studied  as  a  bona  fide  system.  The 
complex  interactions  between  the  factors  have  made  it  difficult  to 
gain  meaningful  insights  from  the  uni-variate  studies  so  common  in 
maintenance  studies.  Consequently,  little  is  known  about  the 
sensitivity  of  system  performance  parameters  to  changes  of  the 
factors.  To  compound  the  problem,  some  variables  affect  not  only  the 
system  performance  parameters  but  the  demand  for  other  factors  as 
well.  For  instance,  a  reduction  in  human  errors  that  damage 
components  should  increase  operational  readiness.  However,  it  should 
also  reduce  requirements  for  such  factors  as  spare  parts,  maintenance 
facilities,  and  personnel. 

The  Random  Nature  of  Aircraft  Component  Failure 

The  primary  reason  why  maintenance  as  a  system  is  so  complex  is  that 
it  is  a  stochastic  system.  That  is,  it  is  a  system  "driven"  by  random 
events.  The  random  events  driving  the  system  are  the  failures  of  the 
system  components.  The  maintenance  system  is  "driven"  by  the  failures 
in  that  it  must  respond  to  the  failures  and  correct  the  malfunctions. 

The  random  distribution  of  the  failures  complicates  the  system  in  two 
ways.  First,  the  specific  time  of  failure  cannot  be  predicted.  The 
only  prediction  possible  is  the  total  number  of  failures  over  an 
expected  period  of  operation.  Thus,  resources  for  corrective 
maintenance  cannot  be  scheduled  in  advance. 

Secondly,  the  random  distribution  usually  results  in  work  demands 
occurring  in  batches.  That  is,  the  average  time  between  failures  for 
a  given  component  may  be  long  but  the  actual  time  between  two  failures 
may  be  quite  short,  followed  by  a  very  long  interval  for  the  next 
failure.  When  hundreds  of  components  are  involved,  the  slack  and  busy 
times  become  mixed  together,  thus,  the  "balancing"  of  resource 
utilization  and  queues  becomes  quite  complex. 

ALTERNATE  APPROACH:  A  COMPUTER  MCDEL 

The  two  properties  discussed  above  make  it  quite  difficult  to  learn 
the  characteristics  of  a  maintenance  system  by  systematically 
observing  an  actual  maintenance  operation.  Such  systematic 
observations  become  quite  time-consuming,  expensive,  and  often 
disruptive. 
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An  alternative  to  the  observation  method  must  be  a  method  that  handles 
the  complex  interactions  properly  and  can  properly  handle  the  random 
occurrences.  The  method  must  also  provide  quantitative  information. 
Computer  modeling,  using  Monte  Carlo  simulation  techniques,  satisfies 
all  of  these  requirements. 

COMPUTER  MODELING  WITH  MONTE  CARLO  SIMULATION 

A  model  is  a  representation  of  a  theory  or  an  empirical  system.  To  be 
useful,  the  way  in  which  the  model  works  must  correspond  to  how  the 
real  system  works.  The  system  to  be  modeled  was  shown  in  graphic  form 
in  Figure  1.  The  model  is  intended  to  represent  the  "aircraft  flight 
and  maintenance  system"  portion.  The  inputs  and  outputs  remain 
basically  the  same.  Given  the  selected  set  of  inputs  in  Figure  1,  the 
model  generates  the  outputs. 

The  Monte  Carlo  simulation  technique  is  especially  suited  for  modeling 
a  maintenance  system.  The  suitability  is  discussed  in  terms  of  the 
two  important  properties  discussed  earlier,  i.e.,  complex  interactions 
and  randomness.  The  unique  feature  of  the  Monte  Carlo  technique,  and 
the  basis  for  its  name,  is  its  ability  to  simulate  randomly  occurring 
events.  Some  precautions  about  the  model  are  also  presented. 

Complex  Interaction 


The  modeling  method  is  well  suited  to  systems  with  many  complex 
interactions.  The  volume  of  information  necessary  to  handle  all  of 
the  many  details  of  the  interactions  requires  that  the  model  be 
implemented  on  a  computer. 

For  a  squadron  of  12  aircraft,  each  with  more  than  100  units,  and 
approximately  a  hundred  men  distributed  over  11  work  centers,  the 
computer  can  simulate  a  month  of  flights  and  maintenance  in  a  few 
seconds.  The  several  million  calculations  and  manipulations  that  a 
month  of  simulation  represents  could  not  be  accomplished  in  any  other 
way. 

Randomness 


Because  aircraft  failures  occur  randomly,  the  computer  model  must  be 
able  to  simulate  such  random  failures.  The  Monte  Carlo  technique 
generates  randomly  occurring  failures. 
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In  the  Monte  Carlo  technique,  the  probability  of  component  failure 
must  be  provided  as  an  input.  This  input  is  expressed  as  the 
probability  that  a  particular  component  will  fail  per  flight  hour.  To 
determine,  on  a  particular  occasion,  if  the  component  fails,  the 
computer  selects  a  random  number  from  a  long  list  of  random  numbers 
between  zero  and  one. 

For  example,  the  probability  that  a  coin  flipped  will  be  "heads"  is 
0.5.  To  simulate  the  flip  of  a  coin,  the  computer  selects  a  number 
from  a  list  of  random  numbers  between  0  and  1.  If  the  random  number 
is  less  than  .5000,  the  computer  treats  the  result  of  the  coin  flip  as 
a  head.  Otherwise,  the  result  is  treated  as  a  tail.  The  example 
simulates  a  random  coin  flip  using  Monte  Carlo  techniques. 

In  a  similar  way,  the  model  simulates  random  failures,  using  the 
probability  of  failure  per  flight  hour  for  each  of  the  components  on 
the  aircraft.  The  probabilities  currently  in  the  model  are  based  on 
the  actual  failure  rates  for  the  F-14A  units.  Let  us  assume  that  the 
probability  that  a  particular  unit  will  fail  in  a  given  hour  of  flight 
is  15  percent  or  0.15.  After  an  hour  of  simulated  flight,  the 
computer  selects  a  random  number  from  the  random  number  list. 

If  the  random  number  is  less  than  0.15,  the  unit  is  considered 
to  be  a  failed  component.  If  it  is  greater,  the  unit  is  still 
considered  good.  After  each  hour  of  flight,  the  next  random  num¬ 
ber  in  the  list  of  random  numbers  is  selected. 

This  approach  has  two  advantages.  For  one,  it  provides  the  randomness 
required  by  actual  aircraft  failure  rates.  Just  as  important,  it  also 
allows  a  repeatability  not  possible  in  the  real  world.  For  instance, 
by  starting  the  computer  again,  at  the  beginning  of  the  random  number 
list,  one  can  repeat  exactly  each  component  failure  at  identical  times 
in  the  run.  Thus,  the  computer  run  may  be  repeated  exactly  with  an 
identical  number,  order,  and  timing  of  failures. 

Obviously  there  is  no  advantage  to  repeating  a  run  if  nothing  is 
changed.  However,  there  are  times  when  it  would  be  of  tremedous 
advantage  to  repeat  a  run  or  portion  thereof  and  examine  the  run  in 
greater  detail.  This  is  particularly  useful  when  trying  to  isolate 
problems  with  model  operation. 

EXie  to  the  randomness  of  failures,  the  wee k-to -week  or  month- to-month 
variation  of  system  performance  parameters  is  very  high.  This  means 
that  an  average  of  many  measurements  taken  during  a  reasonable  period 
of  time  is  needed  to  obtain  a  representative  figure.  Usually,  this 
period  is  six  months  or  more. 
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This  high  variance  makes  it  impractical  to  study  the  sensitivity  of 
the  system  to  specific  changes  by  using  the  actual  system.  With  the 
model,  six  months  of  actual  time  can  be  simulated  in  about  a  minute  of 
computer  time.  Thus,  the  model  can  be  used  to  test  the  potential  of 
proposed  changes  before  actually  implementing  such  changes.  In 
addition,  model  runs,  with  systematic  changes  of  the  factors,  can 
determine  the  extent  to  which  the  system  is  sensitive  to  changes  in 
the  individual  factors. 

PRECAUTIONS 

There  are  several  difficulties  with  models.  Most  systems  worthy  of 
modeling  are  very  complex.  In  comparison,  the  AMES  type  of  model  is 
quite  simple.  In  fact,  the  strength  of  the  model  lies  in  its 
simplicity.  However,  the  simplicity  is  also  its  point  of  vulnerability 
—  if  the  model  is  not  used  properly. 

In  reducing  the  complexities  to  the  relatively  simple  model,  it  is 
necessary  to  make  abstractions  and  take  shortcuts.  For  example,  an 
actual  aircraft  has  millions  of  components.  In  a  model,  these 
components  are  grouped  together  into  a  few  hundred  units  or  about  a 
hundred  subsystems.  The  variations  in  failures,  errors,  time, 
maintenance  manhours,  etc.,  between  components  (within  a  subsystem) 
are  "treated"  with  a  distribution. 

A  similar  reduction  process  is  applied  to  the  other  system  resources. 
For  example,  human  performance  is  a  highly  complex  process.  Yet,  the 
process  is  represented  by  only  three  parameters,  i.e.,  time,  error 
rate,  and  number  of  personnel  (per  job). 

These  simplifications  should  not  be  a  problem  if  the  user  recognizes 
the  level  at  which  the  system  and  its  resources  are  represented.  For 
example,  the  model  cannot  help  determine  the  effect  of  stress  on  an 
individual  technician.  However,  the  model  can  help  determine  the 
effect  of  stress  on  system  performance  if  the  stress  can  be  expressed 
in  terms  of  some  change  in  performance  of  technicians. 
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SECTION  IV 

DESIGN,  CONSTRUCTION,  AND  TESTING  OP  TOE  AMES  MODEL 


The  implementation  of  complex  computer  models  is  a  large  effort. 
Choosing  the  proper  technique  saves  time,  and  produces  models  that  are 
easier  to  understand,  build,  and  fix.  This  section  describes  the 
considerations  and  procedures  used  to  design,  construct,  and  test  the 
AMES  Model. 

INTRODUCTION 

The  principal  difficulty  encountered  in  building  a  model  is 
communication  —  both  verbal  communication  and  the  ability  to 
understand  various  documents  produced.  The  information  required  to 
design  and  construct  the  AMES  Model  is  distributed  among  different 
people  as  follows: 

Naval  Aircraft  Maintenance  Experts  -  They  have  an  understanding  of  how 
the  maintenance  system  operates.  Their  direct  knowledge  of 
procedures,  decision  criteria,  and  priorities  must  be  reflected  in  the 
workings  of  the  model.  The  maintenance  experts  have  a  knowledge  of 
available  3-M  statistics  and  what  the  statistics  indicate.  Because 
these  experts  are  also  potential  users,  the  outputs  should  be 
understandable  to  them. 

NAVTRAEQUPCEN  Personnel  -  The  project  is  funded  and  technically 
monitored  through  NAVTRAEQUIPCEN.  They  must  be  able  to  under¬ 
stand  how  the  project  has  proceeded  and  what  the  model  does. 

Model  Designers  -  The  model  designers  have  a  knowledge  of  the  pitfalls 
and  capabilities  of  models.  In  order  to  complete  the  design  they  need 
to  understand  the  nature  of  aircraft  maintenance,  the  nature  of  3-M 
statistics,  and  the  nature  of  model  trade-offs. 

Model  Programmers  and  Testers  -  These  people  may  occasionally  be  those 
who  design  the  model.  When  they  are  not,  they  need  an  understanding 
of  design  as  well  as  knowledge  of  the  computer  language  required  to 
implement  the  model. 
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Model  Modifier  -  Frequently  in  the  life  cycle  of  a  computer  program, 
requirements  of  the  model  change,  either  because  demands  on  the  model 
change  or  the  modeled  maintenance  procedures  are  altered.  As  model 
usage  is  extended  to  new  situations,  new  bugs  invariably  are 
discovered.  Model  improvements  are  discovered  as  the  model  is  used. 
Model  modification  requires  that  the  model  be  easy  to  follow.  A 
knowledge  of  computers  and  of  the  proper  languages  is  necessary; 
however,  no  other  knowledge  should  be  necessary  to  quickly  understand 
and  alter  the  model. 

Model  Users  -  The  users  of  the  model  represent  the  most  diverse  group. 
It  is  anticipated  that  users  will  be  either  researchers  or  maintenance 
personnel . 

The  wide  variety  of  background  and  training  among  those  involved  means 
that  techniques  to  facilitate  communication  and  understanding  are 
essential.  This  subsection  covers  design,  construction  and/or  testing 
techniques  under  the  following  additional  headings: 

•  Design  and  Construction  of  the  Model 

•  Design  of  the  Model  Outputs 

•  Testing  and  Validation 

The  first  subsection  discusses  techniques  and  methods  used  to  create 
the  computer  model.  Techniques  of  documentation  and  communication  are 
also  discussed. 

The  second  subsection  discusses  consideration  given  to  the  outputs  of 
the  model.  The  outputs  vary  depending  on  the  intended  use  and 
audience.  The  outputs,  with  only  slight  changes,  may  be  useful  for 
entirely  different  applications,  e.g.,  research  or  training. 

The  third  subsection  reviews  the  procedures  used  to  test  the  model  and 
assure  its  validity.  A  discussion  of  the  meaning  of  testing  and 
validation  is  included. 

DESIGN  AND  CONSTRUCTION  OF  THE  MCDEL 

The  subsection  discusses  techniques  and  procedures  to  aid  in 
conmunication,  documentation,  design,  or  construction.  The  following 
are  discussed: 
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•  English  language  design 

•  Software  Design  and  Docunentation  Language  (SDDL) 

•  lb p- down  design 

•  Modular  design 

•  Structural  language  -  Simscript  II. 5 

Each  of  the  above  techniques  or  procedures  is  described  below. 

English  Language  Design 

To  satisfy  the  need  for  communication  among  people  of  different 
backgrounds,  the  language  used  for  the  design  of  the  model  is  English. 
English  is  readily  readable  by  all  those  involved  with  the  project. 
The  English  description  is  collected  into  small  modules  so  it  is 
structured  very  much  like  a  computer  program. 

Designing  in  English  has  the  following  advantages: 

•  Facilitates  communication  with  maintenance  personnel  to  allow 
feedback  from  model  designers. 

•  Permits  simpler  understanding  of  the  details  of  the  model  by 
managers  or  users  of  the  model. 

•  Simplifies  efforts  to  modify  and  repair  the  model. 

Software  Design  and  Documentation  Language  (SDDL) 

The  English  language  design  formulated  in  small  modules  can  be  indexed 
and  cross-referenced  using  a  computer  language  processor  called 
Software  Design  and  Documentation  Language  (SDDL).  SDDL  was  written 
by  Henry  Kleine.  The  design  and  the  indexed  cross-reference  are 
combined  to  form  the  Design  Document.  The  Design  Document  is  the 
intermediate  between  spoken  English  and  the  computer  code.  It  is 
English-like,  yet,  because  it  is  indexed  and  cross-referenced,  it  is 
useful  for  both  the  design  phase  and  the  eventual  translation  of  the 
design  into  computer  language. 

Top-Down  Design 

"Top-down"  design  is  the  division  of  the  model  into  several  levels  of 
detail.  First  a  top  level  (without  detail)  design  is  produced.  At 
each  succeeding  level  more  detail  is  included. 
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Top-down  design  is  primarily  a  design  tool.  It  prevents  the  important 
concepts  from  becoming  lost  in  the  mass  of  details.  Problems  that 
arise  in  the  design  can  be  corrected  at  the  highest  level  first. 
After  designing  the  program  from  the  top  down,  translation  into 
computer  language  proceeds  similarly.  Serious  flaws  can  be  cured 
while  the  model  is  relatively  simple.  Frequently,  with  top-down 
design,  as  the  complexity  of  the  program  grows,  the  magnitude  of  the 
errors  is  diminished. 

A  second  benefit  of  top-down  design  is  to  explain  the  workings  of  the 
model.  It  allows  those  learning  about  the  model  to  understand  one 
level  at  a  time.  Since  few  want  to  understand  all  the  details,  when 
the  desired  level  of  detail  is  reached,  one  need  not  continue.  (The 
description  of  the  model  in  Appendix  A  is  constructed  in  a  top-down 
manner.) 

A  final  benefit  of  top-down  design  lies  in  modification  of  the 
program.  Selection  of  the  areas  that  must  be  modified  proceeds  at  the 
highest  levels  of  detail  (very  general).  When  the  section  that 
requires  modification  is  pinpointed,  it  can  be  studied  selectively  in 
much  more  detail. 

Modular  Design 

The  human  mind  cannot  grasp  large  amounts  of  knowledge  easily.  To 
avoid  this  problem,  the  design  (and  eventual  program)  is  composed  of 
many  small  (about  1  page  maximum)  modules.  Each  is,  in  so  far  as 
possible,  self-contained.  Those  parameters  that  must  be  exchanged 
between  these  modules  are  specified.  At  each  level  of  increasing 
detail,  each  module  is  broken  into  several  more  detailed  modules.  The 
modularity  of  the  design  foreshadows  the  eventual  modularity  of  the 
computer  program.  When  the  program  is  written,  each  design  module 
becomes  a  computer  program.  Modularity  also  simplifies  modification 
and  testing  because  problems  frequently  can  be  isolated  to  a  single 
module. 

Structured  Language  -  Simscript  II. 5 

Modern  computer  languages  are  designed  using  new  techniques  to  speed 
ip  the  programing  process.  These  languages  are  foind  to  be  easier  to 
use,  understand,  and  modify.  Such  languages  are  referred  to  as 
structured.  The  language  chosen  for  the  implementation  of  the  model 
is  Simscript  II.  5.  Simscript  II. 5  is  a  modern  language.  Its 
programing  elements  are  structured. 
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As  an  additional  advantage,  Simscript  II. 5  has  been  designed  to  be  as 
close  to  English  as  possible.  The  actual  computer  language  code  in 
most  cases  reads  like  English.  Since  the  names  in  the  design  docunent 
are  the  same  as  in  the  coded  Simscript  II. 5  statements,  many  times  the 
actual  computer  instruction  can  be  understood  by  one  with  no  computer 
experience. 

Finally,  the  combination  of  SDDL  and  Simscript  II. 5  is  ideal:  the 
indexing  and  cross-referencing  feature  of  SDDL  can  also  be  used  on  the 
computer  program.  Having  a  cross-reference  and  index  for  the  actual 
program  simplifies  repair  and  modification. 

DESIGN  OF  THE  OUTPUTS  OF  THE  MODEL 

The  outputs  of  the  computer  model  perform  two  functions:  they  present 
results  to  the  user,  and  they  aid  those  who  test  the  model.  For  much 
of  the  presentation  bo  the  user,  only  general  outputs  are  necessary. 
For  most  uses  the  monthly  average  percentages  of  operational  readiness 
and  missions  completed  are  the  only  necessary  outputs.  If  monthly 
status  reports  are  added,  the  general  outputs  amount  to  only  several 
pages  of  data  for  each  month  of  simulated  time. 

In  contrast,  very  detailed  specific  outputs  are  necessary  for  most  of 
the  tests.  For  example,  the  aircraft  number,  time  of  day,  and  total 
flight  hours  to  date  for  each  takeoff  and  landing,  might  be  required 
for  testing.  These  detailed  outputs  require  several  hundred  pages  of 
output  per  month.  This  amount  of  output  is  expensive,  and  tedious  to 
go  through. 

For  the  two  separate  needs,  (presentation  to  the  user  and  testing  of 
the  model)  ,  two  different  kinds  of  output  are  required.  These  are 
permanent  outputs  and  temporary  outputs  (traces) .  This  subsection 
describes  the  two  different  types  of  outputs  as  follows: 

•  Design  of  Permanent  Outputs 

•  Design  of  the  Trace  Outputs 

The  first  describes  the  permanent  general  outputs.  The  second 
describes  the  detailed  outputs  which  can  be  selected  before  each  run. 
These  detailed  outputs  which  can  be  selected  are  called  traces. 

The  outputs  of  the  model  do  not  affect  its  operation.  Thus,  new 
outputs  giving  additional  information  are  easily  added. 
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Design  of  Permanent  Outputs 

Permanent  outputs  will  be  used  by  persons  who  may  not  be  familiar  with 
model  concepts.  Thus,  the  permanent  output  reports  must  be  easy  to 
understand. 

The  form  of  the  3-M  summaries  has  been  chosen  as  a  prototype  for  the 
more  complicated  suitmaries  that  the  model  produces.  This  assures  that 
those  familiar  with  aircraft  maintenance  can  easily  understand  model 
outputs.  Comparison  of  the  outputs  of  the  model  with  actual  3-M 
summaries  is  facilitated. 

The  following  reports  are  available  whenever  the  model  is  run: 

•  Monthly  Readiness  Reporting  Status  (RRS)  Report  by  Aircraft 

•  History  of  RRS  (12  month) 

•  Manhour  Summary 

•  Additional  Reports 

Monthly  Readiness  Reporting  Status  (RRS)  Report  by  Aircraft  -  A 
typical  example  of  the  monthly  RRS  Report  is  shown  in  Figure  2.  The 
report  includes  all  categories  on  the  corresponding  3-M  report  (CNAP 
Gen  Form  4790/6)  except  for  Reduced  Material  Conditions  because  of 
scheduled  maintenance.  Entries  to  this  column  can  never  occur; 
therefore  the  column  was  deleted. 

The  entries  in  the  row  labeled  TOT  are  the  totals  for  all  of  the 
aircraft.  With  the  exception  of  the  NrM  percentages,  the  entries  in 
the  row  labeled  PCT  are  the  ratio  of  the  column  totals  to  the  total 
HRS  IN  RRS  (column  2) .  The  AW-NOR  is  the  ratio  of  the  total  AtoM-NOR 
to  the  TQTAL-NOR.  The  AWM-RMC  is  the  ratio  of  the  total  AWM-RMC  to 
the  sum  of  the  total  UNSKED-RMC  and  the  total  NFE-RMC.  These  AWM 
percentages  are  respectively  the  percentage  of  the  NOR  time  or  RMC 
time  spent  waiting  for  maintenance  personnel  or  facilities. 

History  of  RRS  -  (12  month)  -  A  yearly  summary  of  the  important 
parameters  of  the  RRS  is  shown  in  Figure  3.  This  report  contains  RRS 
percentages,  total  flights,  and  total  flight  hours  during  the  period, 
as  well  as  aircraft  utilization. 
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Figure  2.  Example  Of  Monthly  RRS  Report 
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Figure  3.  Example  Of  Twelve  Month  History  Of  RRS 
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The  percentages  are  taken  from  the  bottom  of  the  RRS  report  for  each 
of  the  preceding  12  reports.  Aircraft  utilization  is  the  average 
number  of  flight  hours  per  aircraft  during  the  period.  The  column 
labeled  12  above  the  dashed  line  is  the  most  recent  report.  This 
yearly  summary  presents  information  found  in  two  3-M  reports  (CNAP  Gen 
Forms  4790/7  and  4790/5). 

Manhour  Summary  -  Hie  manhour  summary  (see  Figure  4)  records  manhour 
statistics  by  manhours  of  maintenance  time  required.  The  manhours  and 
manhours  per  flight  hour  are  given  by  3-M  records  (CNAP  Gen  Form 
4790/11) .  The  manhour  suninary  of  the  model  does  not  go  into  the  same 
detail  with  Preventive  Maintenance  (PM)  as  do  the  3-M  reports.  PM  is 
not  considered  in  great  detail  because  it  is  not  the  focus  of  the 
model. 

The  manhour  summary  also  includes  miscellaneous  information  applicable 
to  model  operation:  number  of  ll/d  errors;  number  of  missions  flown, 
scrubbed  or  deferred;  and  the  averages  of  the  percent  of  time  OR,  and 
percent  of  missions  flown.  The  last  two  are  the  system  outputs 
described  in  Section  III. 

Additional  Reports  -  Two  additional  status  reports  are  the  final 
permanent  reports.  These  two  reports  include  the  total  number  of  jobs 
by  type,  total  elapsed  maintenance  time  (EMT) ,  total  flights,  total 
flight  hours,  and  the  location  of  each  aircraft  at  the  time  of  the 
report.  These  reports  are  useful  for  comparison  when  error  rates  are 
changed.  Figure  5  shows  typical  examples  of  these  reports. 

Design  of  the  Traces 

The  traces  are  outputs  which  are  selected  at  the  time  the  simulation 
is  begun.  At  present,  the  traces  are  planned  only  for  testing.  For 
future  training,  such  traces  are  useful  for  examining  in  detail  the 
workings  of  the  model.  A  typical  example  is  shown  in  Figure  6.  The 
figure  shows  model  outputs  with  only  pure  traces  selected  (takeoffs, 
landings,  aircraft  movement,  aircraft  NOR  job  report,  and  mission 
deferral).  The  model  offers  more  than  30  traces.  These  traces 
include  detailed  information  on  inputs,  CM  jobs,  inspection  reports, 
component  failures,  etc.  The  traces  are  essential  for  model  testing. 
Traces  are  discussed  further  in  the  following  subsection. 
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TESTING  AND  VALIDATION 


Testing  is  a  process  to  expose  problems  with  the  model.  When  the 
testing  has  been  completed,  validation  of  the  completed  model  can 
proceed.  Validation  is  a  check  of  the  consistency  of  the  model 
outputs  with  actual  flight  records.  Before  the  model  can  be  trusted 
as  a  predictor,  it  must  be  shown  to  reproduce  relevant  features  of 
known  results. 

This  sub-section  describes  briefly  some  of  the  procedures  used  in 
testing  the  model : 

•  Top-down  testing 

•  Manipulation  of  input  data 

•  The  use  of  traces 

Top-Down  Testing 

Top-down  testing  follows  from  top-down  design.  In  top-down  design 
(described  earlier  in  this  section  and  demonstrated  in  Appendix  A)  , 
the  important  general  features  are  designed  first.  Details  are  added 
in  successive  stages  later.  Top-down  testing  proceeds  parallel  to  the 
design  as  the  design  is  implemented. 

The  most  important  advantage  of  top  down  testing  is  that  the  large 
mistakes  that  affect  most  of  the  model  are  discovered  at  an  early 
stage  in  program  development.  As  implementation  and  testing  of  the 
model  proceed,  the  detected  mistakes  are  smaller,  hence  easier  to 
correct. 


Manipulation  of  Input  Data 


While  the  top-down  testing  explained  in  the  last  two  paragraphs 
provides  the  basic  testing  philosophy,  most  of  the  actual  errors  are 
detected  by  systematically  varying  the  input  data.  Systematically 
changing  the  inputs  changes  the  output.  Only  in  rare  cases  is  the 
output  known  for  a  given  set  of  inputs,  therefore,  the  output  must  be 
estimated.  The  following  techniques  to  estimate  change  in  output  for 
a  given  input  change  have  proven  helpful: 

•  Using  simple  data 

•  Using  small  variations 

•  Using  extreme  data 

•  Using  known  data  (validation) 


Using  Simple  Data  -  It  is  very  difficult  to 
with  hundreds  of  components  and  several 
However,  it  is  often  possible  to  understand 
only  two  components  and  two  maintenance 
using  very  simple  data,  in  combination  with 
many  problems  can  be  isolated  and  corrected 


estimate  how  12  aircraft 
hundred  men  will  interact, 
how  a  single  aircraft  with 
personnel  will  react.  By 
the  following  techniques, 
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Using  Small  Variations  -  The  purpose  of  these  techniques  is  to  predict 
specific  levels  of  an  output.  If  such  a  prediction  is  not  possible,  a 
reasonable  substitute  is  to  predict  general  changes  of  an  output  with 
a  controlled  change  of  input.  By  changing  only  a  single  parameter,  a 
qualitative  change  in  output  can  often  be  estimated  and  used  as  a 
basis  for  determining  gross  errors. 

For  example,  suppose  manpower  is  the  only  input  parameter 
changed.  It  is  reasonable  to  assume  that  a  10  percent 
increase  in  personnel  should  probably  increase  operational 
readiness  by  only  a  small  amount.  Therefore,  model  outputs 
showing  a  large  change  in  operational  readiness,  with  a  10 
percent  change  in  personnel  suggest  probable  errors  in  the 
program.  The  errors  can  be  isolated  by  further  systematic 
changes  in  the  input . 

Using  Extreme  Data  -  Occasionally,  making  very  large  changes  in  the 
input  produces  output  values  that  can  be  estimated.  Input  data  are 
considered  extreme  if  they  are  so  large  (or  so  small)  that  further 
changes  will  not  affect  the  output.  In  many  cases,  extreme  data 
eliminate  complexity  in  the  model.  In  the  simpler  extreme  case, 
outputs  can  be  predicted  and  compared  with  actual  model  results. 

Extreme  data  can  be  entered  for  the  number  of  personnel  at  the  work 
centers.  If  there  are  no  people  assigned  to  any  of  the  work  centers, 
there  should  be  no  maintenance  work  done.  The  other  extreme  is  200 
men  at  each  work  center.  Assigning  200  men  to  each  work  center 
(normally  there  are  2  to  10  men)  assures  that  maintenance  personnel 
will  always  be  available.  With  so  many  people  at  a  work  center,  no 
delays  should  be  caused  by  lack  of  maintenance  personnel. 

Using  Known  Data  (Validation)  -  Using  data  from  actual  statistics  is 
validation.  In  addition  to  normal  validation  procedures,  actual 
statistics  can  serve  as  a  baseline  from  which  to  use  the  techniques 
previously  mentioned:  small  variations  and  extreme  data. 

The  Use  of  Traces 


Normally  only  the  output  reports  are  produced  by  a  model  run.  More 
detailed  descriptions  of  the  workings  of  the  model  are  occasionally 
desirable  to  check  out  the  model.  Earlier  in  this  section,  the  nature 
of  the  trace  outputs  was  described.  The  property  that  makes  traces 
useful  for  testing  is  the  ability  to  duplicate  runs  exactly,  including 
random  failure  (described  in  Section  III). 
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When  additional  detail  is  required,  the  identical  simulation  can  be 
rerun,  obtaining  more  detailed  output.  If,  for  example,  a  particular 
month  has  very  few  flight  hours,  the  same  simulation  can  be  rerun,  but 
with  output  information  on  the  times  of  all  take  offs  and  landings. 
Examination  of  the  outputs  at  the  more  detailed  level  facilitates 
isolation  of  errors  or  anomalies. 

c 

The  types  of  information  available  using  traces  include  detailed 
information  on  the  following: 

•  jobs  (start-time,  duration) 

•  manpower  requirements  and  allocation 

•  inspection  results  and  component  failures 

•  PM  scheduling  and  requirements 

•  model  inputs 

•  mission  scheduling 

The  traces  can  be  indexed  by  aircraft  or  job.  Histories  of  a  single 
aircraft  or  a  single  job  can  be  followed  in  great  detail  during  a 
model  run. 
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SECTION  V 

INPUTS  REQUIRED  FOR  THE  MODEL 


INTRODUCTION 

Before  the  simulation  is  run,  details  of  the  aircraft,  maintenance 
personnel,  planned  maintenance,  etc.,  must  be  supplied  to  the 
computer.  Input  data  typical  of  actual  flight  and  maintenance 
conditions  are  used  for  baseline  studies  and  to  check  out  the  model. 
These  typical  data  have  been  derived  from  analysis  of  Naval  3-M 
reports.  Model  runs  with  data  from  actual  flight  statistics  provide  a 
frame  of  reference.  Subsequent  runs  with  systematic  changes  of  input 
parameters  provide  insight  into  the  maintenance  process  and  indicate 
quantitatively  the  effect  of  the  hypothetical  input  variations. 

The  discussion  of  inputs  in  this  section  is  divided  into  the  following 
subsections: 

•  Model  Input  Requirements 

•  Data  Sources 

•  Estimation  of  Corrective  Maintenance  Error  Rates 

•  Generating  Component  Failures 

The  first  subsection.  Model  Input  Requirements,  includes  detailed 
information  about  the  nature  of  the  inputs  and  about  how  the  inputs 
are  derived. 

The  second  subsection.  Data  Sources,  describes  the  sources  used  to 
discover  proper  inputs.  These  sources  of  the  input  data  frequently  do 
not  give  the  precise  information  required  by  the  model.  Therefore, 
the  relevant  information  must  be  extracted  from  available  statistics. 

The  third  subsection,  Estimation  of  Corrective  Maintenance  Error 
Rates,  shows  how  quantitative  values  were  obtained  for  the  errors  made 
by  personnel  involved  in  organizational  maintenance.  The  majority  of 
these  errors  are  not  explicitly  indicated  in  3-M  statistics.  They 
must  be  inferred. 

The  fourth  subsection.  Generating  Component  Failures,  discusses  how 
component  failures  are  generated  by  the  model.  Using  the  error  rates 
explained  in  the  previous  subsection,  component  failures  can  be 
separated  into  those  caused  by  actual  component  failure  and  those 
caused  by  hunan  error. 


41 


NAVTRAEQUI PCEN  77-D-0028-1 
MODEL  INPUT  REQUIREMENTS 

In  order  to  set  up  and  execute  a  simulation  run,  the  model  program 
requires  six  types  of  input  data.  The  model  program  requires  that 
each  data  type  be  represented  by  specific  characteristics  with  each 
characteristic  in  a  specific  form. 

There  are  four  specified  forms  of  input  data.  Each  unit  of  data  must 
take  one  of  the  following  forms: 

•  Integers  -  integers  are  vhole  numbers  (no  fractional  part)  .  They 
are  used  to  describe  a  particular  quantity  or  identification  number. 
For  example,  _3  maintenance  personnel  and  aircraft  number  124  are 
integers. 

•  Real  numbers  -  the  real  numbers  include  numbers  with  a  decimal 
fraction.  They  may  describe  quantity  or  probability.  Examples  of 
real  numbers  are  12.86  hours  as  the  mean  time  between  failures,  and 
.64  as  the  probability  that  the  failure  of  a  particular  component 
renders  the  aircraft  NOR. 

•  Compound  variables  -  compound  variables  express  the  probabilities 
of  occurrence  when  several  alternatives  are  possible.  An  example  is 
the  probabilities  determining  which  type  of  job  will  be  generated  when 
a  component  fails.  There  are  three  possible  types  of  jobs: 
troubleshoot,  repair- in-place,  and  remove/replace.  The  probability 
that  each  occurs  can  be  input  bo  the  computer  using  a  single  compound 
variable. 

•  Distribution  -  a  distribution  is  a  set  of  values  which  could  be 
displayed  graphically.  A  distribution  expresses  the  possible  value  a 
variable  can  have  as  well  as  the  probability  that  each  of  the  values 
occurs.  Elapsed  maintenance  time  (EMT)  for  each  component  is 
described  by  a  distribution.  The  distribution  of  elapsed  maintenance 
times  for  a  particular  component  gives  the  probability  that  the  EMT 
will  be  between  4  hours  and  4.5  hours,  for  instance.  Distributions 
are  frequently  derived  from  3-M  statistics,  although  nonempirical 
distributions  can  be  used  as  well. 

An  example  of  a  distribution  is  shown  in  Figure  7.  The  figure  shows  a 
cumulative  histogram  of  the  elapsed  maintenance  time  (EMT) .  The 
ordinate  is  the  number  of  hours  required  to  complete  a  job;  the 
abscissa  is  the  probability  that  the  job  could  be  completed  in  less 
than  a  given  time.  Fbr  instance,  in  the  figure,  the  probability  is 
.75  that  a  job  could  be  completed  in  less  than  5.1  hours  (see  point  A 
of  Figure  7).  These  probabilities  are  referred  to  as  cumulative 
because  the  probabilities  for  all  lesser  times,  added  together, 
constitute  the  cumulative  probability.  The  maximum  possible 
cumulative  probability  is  1.00. 
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The  six  types  of  input  data  required  by  the  model  program  to  set  up 
and  execute  a  simulation  run  are  discussed  below.  They  are: 

•  Subsystem  Tata 

•  Gomponent  Data 

•  Manpower  Delta 

•  Planned  Maintenance  Data 

•  Multipliers  Data 

•  Run  Data 

Subsystem  Data 

For  model  runs,  the  computer  requires  three  variables  for  each 
subsystem  of  the  aircraft: 

•  Criticality 

•  Facilities 

•  Facilities  Delay  Time 

The  data  necessary  to  determine  the  values  of  these  variables  for 
actual  flights  are  obtained  from  the  Readiness  Reporting  Status  (RRS) 
statistics  by  Work  Unit  Codes.  The  RRS  is  part  of  the  3-M  Data 
Summary. 

1.  The  first  variable,  CRITICALITY,  expresses  the  probability  that  a 
particular  component  failure  in  the  subsystem  causes  the  aircraft  to 
be  NOR  (not  operationally  ready) .  CRITICALITY  is  a  real  variable. 

CR [TICALITY  is  the  quotient  of  a  fraction  whose  numerator  is  the 
number  of  times  the  aircraft  is  not  operationally  ready  because  of 
unscheduled  maintenance  (sum  of  NOPM-U  and  NORS) .  The  denominator  is 
the  number  of  times  a  readiness  reporting  status  is  recorded  (total 
RRS) .  The  calculated  value  cannot  exceed  1.00.  It  is  rounded  ofl  to 
the  nearest  .01. 

2.  FACILITIES  expresses  the  probabilities  that  facilities  required  to 
perform  maintenance  on  the  subsystem  are  not  available.  Like 
criticality,  it  is  a  real  variable  that  cannot  exceed  1.00  and  rounded 
to  the  nearest  .01. 

T,he  FACILITIES  entry  is  the  quotient  of  a  fraction  whose  numerator  is 
the  number  of  times  the  aircraft  is  awaiting  maintenance  because 
facilities  are  not  available.  ( AWM  codes  1,  2,  and  5).  The 
denominator  is  the  total  RRS.  The  calculated  value  cannot  exceed 
1.00. 
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3.  FACILITIES  DELAY  TIME  expresses  the  probabilities  associated  with 
delay  times  when  facilities  are  not  available.  FACILITIES  DELAY  TIME 
is  a  distribution.  The  distribution  is  formed  by  making  a  graph  of 
cumulative  probabilities  against  delay  time  using  ANM  codes  1,  2,  and 
5  from  the  3-M  summary. 

> 

Component  Data 

A  component  is  a  subsystem  combined  with  a  work  center  (a  technical 
maintenance  specialty  area).  Each  component  requires  the  following  14 
variables  for  definition. 

1.  SUBSYSTEM  is  an  index  number  identifying  the  subsystem  to  which 
the  component  belongs.  Subsystem  is  an  integer  variable. 

2.  WORK  CENTER  is  an  index  number  identifying  the  work  center  that 
maintains  the  component.  Work  center  is  an  integer  variable. 

-3.  RIP  INVENTORY  expresses  the  probabilities  that  a  part  required  for 
a  Repair-In-Place  (RIP)  action  is  not  available  (Missing)  for  at  least 
six  days.  RIP  INVENTORY  is  determined  from  the  3-M  action  codes  for 
repair-in-place  actions:  B,  Q,  and  C.  RIP  INVENTORY  is  determined  by 
dividing  the  number  of  B,  Q,  and  C  actions  to  the  particular  subsystem 
at  a  particular  work  center.  B,  Q,  and  C  are  3-M  action  taken  codes. 
RIP  INVENTORY  is  a  real  variable  given  to  the  nearest  .01. 

4.  RIP  PARTS  DELIVERY  TIME  expresses  the  distribution  of  delays  (in 
days)  required  in  awaiting  parts  for  repair- in-place  actions.  The 
distribution  is  determined  by  compiling  a  histogram  of  times  spent 
awaiting  maintenance  using  the  repair- in-place  action  codes  B,  Qf  and 
C  that  were  discussed  in  connection  with  RIP  INVENTORY. 

5.  HR  INVENTORY  expresses  the  probability  that  a  part  required  for  a 
Remove/Replace  (RR)  action  is  not  available  within  six  days.  It  is 
calculated  in  the  same  manner  as  the  real  variable  RIP  INVENTORY.  The 
necessary  data  are  obtained  from  the  action  taken  codes  for 
remove/ replace  actions. 

6.  RR  PARTS  DELIVERY  TIME  is  a  distribution.  It  is  analogous  to  RIP 
PARTS  DELIVERY  except  R  and  U  codes  are  used  for  remove/replace 
actions. 
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7.  RR  INVENTORY  PART  BAD  expresses  the  probability  that  a  part 
obtained  from  inventory  for  a  remove/ replace  action  is  NOT  BAD  (good) . 
RR  INVENTORY  PART  BAD  is  a  real  variable.  The  value  is  calculated  by 
first  dividing  the  number  of  times  a  bad  part  has  been  encountered  by 
the  total  number  of  remove/ replace  actions  and  then  subtracting  this 
value  from  1.00. 

8.  ACTION  TAKEN  has  three  steps,  designated  "1"  through  "3",  that 
express  the  probabilities  that  a  maintenance  problem  requires  either 
Troubleshooting  (T/S) ,  Repair-In-Place  (RIP) ,  or  Remove/ Replace  (RR) 
actions.  ACTION  TAKEN  is  a  compound  variable. 

The  T/S  probability  is  calculated  by  dividing  the  recorded 
troubleshooting  actions  by  the  sum  of  troubleshooting, 
repa ir- in-place ,  and  remove/ replace  actions  for  a  particular 
component.  The  RIP  probability  is  calculated  by  dividing  the  recorded 
repair- in-place  actions  by  the  same  sum.  The  RR  probability  is 
calculated  by  dividing  the  number  of  remove/ replace  actions  by  the 
sum.  The  sum  of  these  three  probabilities  is  1.00. 

9.  ERROR  I  (unjustified  removal)  expresses  the  probability  of 
committing  a  Type  I  error  for  the  specified  component  in  an  RR  action. 
ERROR  I  is  a  real  variable. 

10.  ERROR  II/D  (undetected  error/damage)  expresses  the  probability  of 
committing  a  Type  Il/d  error  in  any  corrective  maintenance  action. 
ERROR  II/D  is  a  real  variable. 

11.  MEAN  TIME  BETWEEN  FAILURES  (MTBF)  is  the  number  of  flight  hours 
between  component  failures  for  a  given  component.  MTBF  is  a  real 
variable.  The  value  is  the  quotient  of  a  fraction  whose  denominator 
is  the  total  number  of  flight  hours  of  the  sample  used.  The  numerator 
is  the  number  of  Corrective  Maintenance  Actions  (CMAs)  for  the 
component  of  concern  during  the  flight  hours  mentioned  above. 
Failures  include  corrective  maintenance  actions  resulting  from  Type  I 
and  Type  Il/d  errors.  The  total  flight  hours  are  obtained  from  Flight 
Records  Statistics.  For  the  purposes  of  this  report,  MTBF  and  Mean 
Time  Between  Corrective  Maintenance  Actions  (MTBCMA)  may  be  considered 
equivalent. 

The  number  of  actions  resulting  from  Type  ll/d  errors  is  determined  by 
multiplying  the  total  corrective  maintenance  actions  by  the  ll/d  error 
rate.  The  number  of  Type  I  errors  is  obtained  by  multiplying  the 
Remove/Replace  actions  by  the  Type  I  error  rate.  Using  the  number  of 
Type  I  and  Type  ll/d  errors,  the  Mean  Time  Between  Actual  Defects 
(MTBAD)  is  determined  in  the  computer.  The  MTBAD  is  the  MTBF  with 
CMAs  resulting  from  errors  excluded. 
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12.  MEAN  TIME  BETWEEN  NO  DEFECT  (MTBND)  is  the  mean  nunber  of  elapsed 
flight  hours  between  each  two  occurrences  when  the  component  is 
erroneously  declared  bad.  The  value  is  determined  by  dividing  the 
total  nunber  of  flight  hours  by  the  number  of  no  discrepancy  reports 
against  the  component.  MEAN  TIME  BETWEEN  NO  DEFECT  is  a  real 
variable. 

13.  MEAN  TECHNICIANS  is  the  average  number  of  personnel  needed  to 
perform  corrective  maintenance  on  the  component  of  concern.  MEAN 
TECHNICIANS  is  a  real  variable. 

14.  ELAPSED  MAINTENANCE  HOURS  (EMH)  is  a  distribution  of  the  elapsed 
maintenance  times  (in  hours)  for  a  particular  component.  The  values 
are  calculated  from  actual  component  elapsed  maintenance  time  data 
that  has  been  processed  to  obtain  cumulative  percent  of  occurrence 
values  for  increments  of  0.5  hours. 

Manpower  Data 

In  the  model,  personnel -is  represented  as  belonging  to  work  centers. 
A  work  center  identifies  a  group  of  specialized  personnel  responsible 
for  a  designated  area  of  aircraft  maintenance. 

1.  NUMBER  OF  WCRK  CENTERS  is  an  integer  variable. 

2.  NUMBER  OF  SHIFTS  is  the  number  of  different  shifts  of  maintenance 
personnel  that  work  during  a  given  day.  It  is  an  integer  variable. 

3.  LENGTH  OF  SHIFT  is  the  duration  that  each  shift  must  work.  :t  is 
a  real  variable. 

4.  NUMBER  OF  CDIs  is  the  number  of  Collateral  Duty  Inspectors  (GDIs) 
available  to  work  at  a  given  work  center  for  a  given  shift. 

5.  NUMBER  OF  SENIORS  is  the  nunber  of  senior  technicians  available  to 
work  at  a  given  work  center  for  a  given  shift. 

6.  NUMBER  OF  JUNIORS  is  the  number  of  junior  technicians  available  to 
work  at  a  given  work  center  for  a  given  shift. 


NAVTRAEQU I PCEN  77-D-0028-1 


Planned  Maintenance  Data 

Planned  Maintenance  (PM)  data  cover  the  planned  maintenance  actions 
that  must  be  performed  on  a  regular  basis.  These  actions  are  included 
in  the  model  because  they  impose  a  significant  workload  on  the  work 
centers. 

The  model  performs  the  following  six  types  of  planned  maintenance: 

•  Special  Inspection  -  determined  by  flight  hours 

•  Special  Inspection  -  determined  by  calendar  time 

•  Special  Inspection  -  determined  by  flights 

•  Phased  Inspection  -  determined  by  flight  hours 

•  Turnaround  Inspection  -  determined  by  flights 

•  Daily  Inspection  -  determined  by  calendar  time  (may  also  depend  on 
flight  hours  and  occurrence  of  maintenance) 

The  model  identifies  the  different  planned  maintenance  requirements  by 
performance  intervals.  Each  planned  maintenance  requirement  is 
divided  into  several  segments.  Each  segment  requires  a  specified 
amount  of  time  for  the  work  to  be  performed.  The  work  done  in  a 
segment  may  require  work  at  several  work  centers.  Before  a  new 
segment  can  be  started,  work  at  all  of  the  work  centers  of  the 
previous  segment  must  be  completed.  The  following  characteristics  are 
required  to  describe  each  PM  requirement: 

J .  PM  TYPE  designates  which  of  the  six  types  of  PM  mentioned  above 
applies;  e.g..  Daily  Inspection.  A  number  is  assigned  to  each  of  the 
:;ix  types.  PM  TYPE  is  an  integer  variable.  Although  PM  TYPE  is  not 
actually  input  to  the  computer,  the  information  is  required  to  group 
the  PM  requirements  by  type. 

NUMBER  SEGMENTS  is  the  number  of  different  segments  in  a  PM 
requirement.  NUMBER  OF  SEGMENTS  is  an  integer  variable. 

2.  SEGMENT  TIME  is  the  time  (in  minutes)  required  to  complete  each  of 
the  segments.  SEGMENT  TIME  is  a  real  variable.  Each  of  the  segments 
has  a  SEGMENT  TIME. 

4.  NUMBER  OF  WORK  CENTERS  for  each  segment  is  the  number  of  work 
centers  vhere  work  must  be  done  during  the  segment.  NUMBER  OF  WORK 
CENTERS  is  an  integer  variable. 
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5.  PM  WORK  CENTER  is  the  work  center  where  work  must  be  performed  for 
each  of  the  various  jobs  in  a  particular  segment.  PM  WORK  CENTER  is 
an  integer  variable. 

6.  PM  NUMBER  OF  MEN  is  the  number  of  maintenance  personnel  required 
at  a  particular  work  center  for  a  particular  segnent.  PM  NUMBER  OF 
MEM  is  an  integer  variable. 

Multipliers  Data 

Multipliers  can  be  used  to  make  uniform  changes  of  hunan  performance 
variables.  For  each  of  the  components  or  subsystems,  the  chosen 
variable  is  multiplied  by  the  data  multiplier  in  question.  A  data 
multiplier  of  1.00  leaves  values  unchanged.  Ebr  example,  the  Type  I 
error  rate  for  all  components  can  be  reduced  to  one-half  of  its 
original  value  by  setting  the  ERROR  TYPE  I  MULTIPLIER  equal  to 
one-half.  For  each  variable,  a  data  multiplier  will  modify  the 
associated  variable  for  all  components  in  a  uniform  manner.  The 
following  multipliers  are  used  for  data  inputs: 

•  1CRR0R  TYPE  I  MULTIPLIER 

•  ERROR  TYPE  II/D  MULTIPLIER 

•  'LAPSED  MAINTENANCE  HOURS  (EM  HOURS)  MULTIPLIER 

•  ’ARTS  BAD  FROM  SUPPLY  MULTIPLIER 

•  DEFECT  (UNDISCOVERED  PILOT  SQUAWKS)  MULTIPLIER 

The  multipliers  can  be  used  to  reduce  or  increase  any  of  the  above 
parameters  for  all  of  the  components.  They  are  real  variables. 

Run  Data 


The  run  data  include  other  miscellaneous  entries  such  as  mission 
schedule,  report  intervals,  traces,  and  RUN  DURATION.  Mission  schedule 
includes  a  MISSION  START  TIME  and  a  MISSION  DURATION  for  each  mission. 
The  report  intervals  are  variables  that  adjust  the  time  between 
reports.  Traces  are  for  testing  and  have  been  previously  discussed. 
RUN  DURATION  is  the  duration  of  the  simulation  in  days. 

Summary 

For  convenient  reference,  the  prominent  data  tyises  and  the  forms  they 
take  are  shown  in  Table  5. 
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FORMS  OF 
INPUT  DATA 


Integer  Variables 


Real  Variables 


Compound  Variable 
Distribution 


TABLE  5.  PROMINENT  DATA  TYPES 


VARIABLE  DATA  TYPE 


Subsystem 

Component 

Work  Center 

Component 

Number  of  Work  Centers 

Manpower 

Number  of  Shifts 

Manpower 

Number  of  CDI's 

Manpower 

Number  of  Seniors 

Manpower 

Number  of  Juniors 

Manpower 

PM  Type 

Planned  Maintenance 

Number  of  Segments 

Planned  Maintenance 

Number  of  Work  Centers 

Planned  Maintenance 

PM  Work  Center 

Planned  Maintenance 

PM  Number  of  Men 

Planned  Maintenance 

Criticality  * 

Subsystem 

Facilities* 

Subsystem 

RIP  Inventory* 

Component 

R  R  Inventory  * 

Component 

RR  Inventory  Part  Bad* 

Component 

Error  1* 

Component 

Error  lt/d* 

Component 

MTBF 

Component 

MTBND 

Component 

Mean  Technicians 

Component 

Length  of  Shift 

Manpower 

Segment  Time 

Planned  Maintenance 

Error  Type  1  Multiplier 

Multiplier 

Error  Type  ll/d  Multiplier 

Multiplier 

Elapsed  Maintenance  Hours 

Multiplier 

Multiplier 

Parts  Bad  From  Supply 

Multiplier 

Multiplier 

No  Defect  Multiplier 

Multiplier 

Action  Taken* 

Component 

Facilities  Delay  Time* 

Subsystem 

RIP  Parts  Delivery  Time* 

Component 

RR  Parts  Delivery  Time* 

Component 

Elapsed  Maintenance  Hours' 

Component 

*  =  probability 
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DATA  SOURCES 

The  Navy  3-M  System  contains  a  Maintenance  Data  Collection  Subsystem 
that  furnishes  statistical  data  products  to  the  Naval  Aviation 
Maintenance  Program  (NAMP) .  Six  categories  of  maintenance  data  are 
collected  and  processed.  Two  of  the  categories.  Maintenance  Data 
Reporting  (MER)  and  Aircraft  Statistical  Data  (ASD) ,  have  information 
useful  to  the  AMES  Project. 

The  collected  data  are  transcribed  onto  specific  electric  accounting 
machine  (EAM)  cards.  These  cards  are  submitted  monthly  to  the 
maintenance  Support  Office  for  updating  of  the  MER  History  File. 

A  brief  description  of  the  selected  data  records  is  presented  below  in 
the  following  sequence: 

•  Support  Action  Record 

•  on  Equipment  Action  Record 

•  Removal  Action  Record 

•  AIMD  Action  Record 

•  Technical  Directive  Compliance  Record 

•  Aircraft  Statistical  Data  Record 

•  Aircraft  Flight  Data  Record 

Support  Action  Record 

This  record  is  Card  Type  01.  It  documents  maintenance  manhours 
expended  on  repetitive  nonrepair  type  maintenance  which  are  not 
recorded  on  the  Visual  Display  System/Maintenance  Action  Form 
(VIDS/MAF) .  The  following  Level  1  information  on  the  record  is  of 
interest  to  the  WES  Project: 

•  Both  the  Support  Code  and  the  Type  Maintenance  Code  show  the 
repetitive  type  maintenance  performed. 

•  rhe  Number  of  Items  Receiving  Maintenance  entry  is  used  to 
determine  hours  per  item. 

•  [he  Work  Center  entry  shows  the  technical  skill  category  that 
accomplished  the  work. 

•  The  Total  Number  of  Hours  Expended  by  Maintenance  Personnel  entry 
shoves  how  many  manhours  were  needed  to  accomplish  the  work. 
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On  Equipment  Action  Record 

This  record  is  Card  Type  11.  It  documents  corrective  and  planned 
maintenance  actions  performed  on  a  specific  aircraft.  The  following 
Level  1  information  on  this  record  is  of  interest  to  the  AMES  Project. 

•  The  Wbrk  Unit  Code  (WUC)  shows  how  often  maintenance  was  performed 
on  a  system,  subsystem  or  component. 

•  The  When  Discovered  Code  shows  how  often  a  need  for  corrective 
maintenance  occurred  when  performing  a  scheduled  maintenance  function. 

•  The  Action  Taken  Code  shows  the  WUCs  requiring  troubleshooting 
and/or  repair.  The  codes  also  show  the  different  repair  actions  such 
as  repair- in-place,  remove/ replace,  cannibalization,  awaiting  parts, 
and  adjustments. 

•  The  Number  of  Items  Receiving  Maintenance  entry  helps  in  evaluating 
actual  number  of  manhours  expended  an  one  item. 

•  Manhours  and  Elapsed  Maintenance  Time  entries  show  information  on 
number  of  personnel  used  and  how  long  it  took  to  accomplish  the 
maintenance  task. 

•  Malfunction  Codes,  in  particular  the  conditional  codes,  help 
identify  errors  committed  at  the  organizational  maintenance  level. 

Removal  Action  Record 


This  record  is  Card  Type  21.  It  documents  the  scheduled  removal 
components  and  the  Level  1  corrective  removal  actions  for  repairable 
components  sent  to  the  supporting  Aircraft  Intermedite  Maintenance 
Department  (AIMD) .  The  maintenance  actions  are  documented  for 
specific  aircraft.  This  record  is  of  interest  because  it  will  provide 
the  corrective  removal  actions  requiring  repair  by  a  supporting  AIMD. 
Scheduled  removal  actions  specify  the  repair  or  disposal  action. 

AIMD  Action  Record 


This  record  is  Card  Type  31.  It  documents  the  work  performed  on 
repairable  components  received  from  the  Organizational  Maintenance 
Departments  supported  by  the  AIMD.  Maintenance  actions  performed  are 
recorded  using  the  same  Job  Control  Number  (JCN)  assigned  by  the 
supported  organization.  This  record  is  of  interest  because  it  will 
provide  what  was  the  final  result  of  the  corrective  organizational 
removal  actions.  Information  provided  will  assist  in  identifying 
organizational  maintenance  errors  and  their  causes. 
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Technical  Directive  Compliance  Record 

This  record  is  Card  Type  41.  It  documents  completed  technical 
directives  issued  for  the  aircraft  and  the  maintenance  performed  to 
accomplish  them.  Technical  directive  requirements  include  all  types 
of  changes  which  affect  the  maintenance  and  operation  of  aircraft 
equipment.  Effect  on  maintenance  work  load  and  effectiveness  measures 
for  the  operating  unit  will  be  provided  by  this  record.  The  following 
Level  1  information  is  of  interest  to  the  AMES  Project. 

•  Work  Unit  Code  identifies  equipment  affected  by  technical 
directives. 

•  Items  Processed,  Manhours,  and  Elapsed  Maintenance  Time  entries 
show  the  number  of  men  assigned  and  elapsed  time  bo  accomplish  the 
technical  directive. 

•  Work  Center  Code  shows  the  technical  skill  category  of  assigned 
personnel . 

Aircraft  Statistical  Data  Record 


This  record  is  Card  Type  71.  It  documents  the  nunber  of  hours  a  Not 
Operationally  ready  (NOR)  or  Reduced  Material  Condition  (RMC)  status 
existed  for  a  specific  aircraft.  Also  shown  are  any  awaiting 
maintenance  time,  the  reason  for  the  wait,  and  the  equipment 
responsible  for  the  NOR  or  RMC  status.  This  record  shows  the 
equipment  items  that  frequently  impact  the  performance  measures  of  an 
operational  unit. 

Aircraft  Flight  Data  Record 

This  record  is  Card  Type  76.  It  documents  the  number  of  flights,  the 
total  flight  hours  and  the  type  and  nunber  of  landings  that  occurred 
on  a  specific  date  for  a  specific  aircraft.  Information  on  this 
record  can  be  used  in  the  development  of  data  required  by  the  Flight 
Orders  input  to  the  WES  Model. 
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ESTIMATION  OF  CORRECTIVE  MAINTENANCE  ERROR  RATES 

Human  errors  are  responsible  for  a  significant  fraction  of 
the  total  number  of  aircraft  corrective  maintenance  actions. 

Our  conservative  estimate  is  that  at  least  20  percent  of  the 
non-cannibalization  corrective  maintenance  actions  on  aircraft 
equipment  are  caused  by  human  errors.  Treatment  of  human 
errors  when  performing  corrective  maintenance  has  been  included 
in  the  AMES  model . 

Before  the  effect  of  maintenance  errors  can  be  studied  with  the  model, 
a  baseline  needs  to  be  established,  i.e.,  the  current  level  of  errors. 

Other  human  performance  variables  such  as  elapsed  maintenance  time 
(EMT)  or  maintenance  manhours  are  determined  routinely  by  the  3-M 
system.  As  part  of  this  project,  corrective  maintenance  error  rates 
have  been  conservatively  determined  at  the  subsystem  level  of  aircraft 
equipment.  Knowledge  of  error  rates  permits  calculation  of 
improvements  in  equipment  performance  that  result  from  a  reduction  in 
human  errors. 

All  of  the  errors  discovered  in  our  analysis  are  errors  that  resulted 
in  i  he  generation  of  additional  jobs.  Errors  which  are  detected  by 
the  CDI  or  the  maintenance  technician  before  completion  of  the  job 
result  in  increased  performance  time.  These  errors  were  not 
detectable  by  the  analysis  methods  used.  Also,  Type  I  errors  during 
Repair-in-Place  (RIP)  actions  cannot  be  detected  with  normal  3-M  data. 

Since  RIP  actions  comprise  over  half  of  all  corrective  maintenance 
actions  (excluding  cannibalization),  the  baseline  for  maintenance 
errors  is  understated.  The  methods  for  determining  error  rates  are 
described  briefly  under  the  following  headings: 

•  Database 

•  Measurement  of  Type  I  errors 

•  Measurement  of  Type  II  and  Type  d  errors 

Database 

Data  collected  and  processed  by  the  3-M  System  is  used  as  the  database 
to  estimate  the  organizational  level  aircraft  maintenance  error  rate. 

3-M  data  records  provide  a  dynamic  history  of  aircraft  maintenance 
actions.  The  maintenance  actions  are  recorded  for  equipment  in  its 
intended  operational  environment  and  controlled  by  the  maintenance 
organization.  Maintenance  actions  are  recorded  daily  as  they  occur 
and  compiled  monthly  for  machine  processing  into  various  reports. 
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The  data  and  reports  are  designed  to  provide  timely  information  to 
Naval  Aviation  Maintenance  Program  managers  in  order  to  evaluate, 
correct  and  improve  maintenance  performance.  A  representative  sample 
of  3— M  data  records  was  obtained  and  subjected  to  careful  analysis  and 
processing. 

To  determine  the  equipment  error  rate  statistic,  we  used  a  sample  of; 
one  year  of  data  records  for  a  specific  type  of  aircraft,  maintained 
and  operated  by  six  organizations  over  the  same  period.  Also,  one  3-M 
report  was  processed  to  determine  the  error  rate  statistic  for  tho 
Type  I  error  category.  Data  records  and  3-M  reports  used  are 
described  in  Table  6.  A  more  complete  description  can  be  found  in  an 
earlier  report.  Acquisition  and  Analysis  of  Navy  3-M  Data. 

Error  Rate  is  defined  as  a  statistical  measure  of  the  occurrence  of 
one  or  more  of  the  defined  error  categories.  It  is  expressed  as  a 
percent  of  corrective  maintenance  actions  (E/CMA)  performed.  Error 
rates  are  grouped  for  Work  Unit  Codes  (WUCs)  at  the  subsystem  level  by 
action  work  center.  For  this  project,  the  subsystem  level  is 
identified  by  the  first  three  digits  of  the  WUC. 

Measurement  of  Type  I  Errors 

The  Type  I  error  rate  for  aircraft  equipment  was  calculated  from  data 
provided  by  3-M  report  MSOD-4790.A2551-01,  No  Defect  Item  Analysis 
Summary.  This  report  indicates  those  items  removed  from  the  aircraft 
that  were  found  to  function  propierly  (No  Defect)  by  intermediate 
maintenance.  The  percentage  of  A799  (No  Defect)  items  reported 
represent  a  portion  of  the  Type  I  error  rate. 

The  percent  No  Defect  items  are  the  declared  bad  but  actually  good 
repairable  equipment  items  removed  by  organizational  maintenance 
personnel,  expressed  as  a  percentage  of  the  unscheduled  repairable 
items  removed.  The  percentage  is  for  all  organizational  maintenance 
work  centers  in  all  organizations  maintaining  the  specified  aircraft. 

The  Type  I  error  rate  calculated  from  this  report  does  not  include 
false  removals  of  nonrepayable  equipment  or  erroneous  repair-in-place 
actions.  The  latter  ommission  is  important  since  repair- in-place 
actions  represent  60-65  percent  ' of  corrective  maintenance 
actions  (excluding  cannibalization  actions) .  The  number  of 
Type  I  errors  noted  would  nearly  double  even  if  errors  dur¬ 
ing  repair-in-place  were  only  one-half  of  the  errors  during 
remove/replace.  Unfortunately,  errors  during  repair-in-place 
cannot  be  detected  with  the  current  3-M  data. 
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CARD  TYPE 


11 


21 


76 


Report 
MSOD4790. 
A2551  01 


TABLE  6.  SELECTED  3-M  DATA  RECORDS  AND  REPORTS 
RECORD  NAME  RECORD  DESCRIPTION 


On  Equipment  (Maintenance) 
Action  Record 


Removal  (Maintenance) 
Action  Record 


Aircraft  Flight  Data  Record 


No  Defect  Item 
Analysis  Summary 


Prepared  from  information  on  the  VIDS/MAF. 
Documents  corrective  and  planned  maintenance 
actions.  Actions  are  documented  for  a  specific 
organization  and  for  a  specific  aircraft. 

Prepared  from  information  on  the  VIDS/MAF. 
Documents  scheduled  component  removal  and 
corrective  maintenance  actions  involving  removal 
of  repairable  components.  Actions  are  documented 
for  a  specific  organization  and  for  a  specific  aircraft. 

Prepared  from  information  on  Part  C  of  Naval 
Aircraft  Flight  Record.  Documents  flight 
information  for  a  specific  day  and  aircraft. 

Includes  number  of  flights  and  total  flight  hours. 

Prepared  from  Card  Code  1 1,  21,  31,  and  34.  The 
report  identifies  the  No  Defect  Actions  for  a  given 
repairable  component  as  a  percentage  of  the  total 
no  defect  actions,  processed  at  intermediate  level 
maintenance  activities. 
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The  report  summarizes  the  percent  of  No  Defect  items  to  the  fourth 
digit  (component  level)  of  the  Work  Unit  Code  (WUC)  for  the  equipment. 
Since  the  AMES  Model  simulates  maintenance  only  to  the  third  digit 
(subsystem  level)  of  the  WUC,  the  report  data  was  converted  to  present 
the  percentage  of  No  Defect  items  at  the  third  digit  level  of  the  WUC. 

Measurement  of  Type  II  and  Type  d  Errors 

The  manifestat  ion  of  both  Type  II  (undiagnosed  failure)  and  Type  d 
(damage)  errors  is  the  same  in  the  3-M  statistics.  Both  error  types 
have  the  same  symptoms.  A  corrected  discrepancy  is  followed  soon  by 
another  discrepancy  of  the  same  or  related  component.  An  engineering 
analysis  of  both  discrepancies  determines  if  an  error  was  made  or  if 
the  two  discrepancies  were  independent.  In  some  error  occurrences. 
Type  II  error  can  easily  be  distinguished  from  a  Type  d  error. 
However,  many  error  occurrences  require  engineering  analysis  to 
differentiate  between  error  types.  Because  Type  II  and  Type  d  errors 
both  have  similar  effects  and  are  treated  in  the  same  way  by  the 
model,  it  was  not  necessary  to  separate  the  two  error  types. 

Figure  8  describes  briefly  the  process  used  to  identify  the  errors. 
The  first  step  (labeled  1)  is  to  choose  a  sample  aircraft.  Recall 
that  the  data  base  contains  maintenance  records  for  six  squadrons,  or 
approximately  72  aircraft.  To  reduce  the  error  analysis  to  a 
manageable  task,  a  minimum  sample  of  24  aircraft  (four  from  each 
squadron)  was  established.  The  aircraft  flight  records  were  examined 
to  eliminate  atypical  aircraft  (those  with  too  few  flights). 

The  remaining  aircraft  were  listed  by  squadron  and  aircraft  bureau 
number  i.e.,  a  number  unique  to  each  aircraft.  Bias  in  selection  of 
sample  aircraft  was  eliminated  by  developing  a  random  selection 
scheme.  A  table  of  random  numbers  was  used  to  rearrange  the  aircraft 
in  each  squadron  in  a  random  sequence.  The  first  four  aircraft  in  the 
random  sequence  were  used  for  error  analysis. 

The  next  step  (labeled  2  in  Figure  8)  was  to  obtain  a  hard  copy  time 
ordered  history  of  the  3-M  flight  and  maintenance  records.  The  flight 
record  information  was  arranged  in  a  column-row  matrix  by  use  of  a 
computer.  The  columns  contain  the  information  on  th<>  flight  record. 
Each  row  (or  3  ine)  represents  one  flight  record  sequenced  by  flight 
date. 

Unscheduled  maintenance  (3-M  type  Maintenance  Code  B)  record 
information  was  also  arranged  in  a  column-row  matrix  by  use  of  a 
computer.  The  columns  contain  selected  information  on  the  maintenance 
record.  Each  row  (or  line)  represents  one  maintenance  record  of  the 
corrective  maintenance  action  (CMA)  performed  to  clear  a  discrepancy 
against  the  aircraft.  The  CMA's  are  listed  in  a  time  ordered  sequence 
of  WUC  numbers,  by  action  date.  The  list  results  in  a  time  ordered 
sequence  of  CMAj  performed  on  each  aircraft  system. 
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Starting  with  the  first  record  for  each  system,  succeeding  records  are 
examined  in  sequence  (step  3  in  Figure  8).  The  flight  list  is  checked 
for  total  number  of  flights  between  action  dates.  If  two  or  fewer 
flights  have  taken  place,  a  possible  Type  II  or  Type  d  error  has 
occurred  (step  4).  Both  OlAs  are  then  analyzed  (step  5)  to  determine 
if  an  error  has  occurred.  Analysis  includes: 

•  Check  Job  Control  Number  (JCN)  date  and  sequence  to  verify  latest 
CMA  occurred  after  previous  CMA  was  cleared  on  action  date. 

•  Determine  if  OlAs  are  related  by  analyzing  information  listed  for 
each  CMA,  (i.e.  equipment  WUC,  manhours,  elapsed  maintenance  time. 
When  Discovered  (WD) ,  Malfunction  (MALF) ,  Action  Taken  (AT),  and  Items 
Processed.)  Check  action  taken  column  of  previous  CMA  for  letter  A 
(no  repair  required).  When  letter  A  is  recorded,  a  possible  D  error 
has  occurred.  Another  D  error  symptom  is  when  CMAs  are  performed  on  a 
system  after  successive  flights  and  then  cease  or  return  to  the  usual 
discrepancy  pattern. 

•  If  not  familiar  with  equipment,  check  the  aircraft  maintenance 
manuals  for  system  theory  and  maintenance  procedures  to  determine  if 
latest  CMA  is  related  (i.e.  functional  dependance,  directly  connected 
or  removed  for  access)  to  previous  CMA.  When  an  error  is  found  (step 
6),  the  CMAs  are  recorded  as  a  D/d  error  (step  7). 

After  analysis  of  all  sample  aircraft  are  complete  (step  8) ,  the 
errors  and  CMAs  are  totalled  and  recorded  for  each  aircraft  equipment 
subsystem.  For  each  equipnent  subsystem,  the  total  errors  are  divided 
by  the  total  CMAs  to  obtain  the  estimate  of  the  subsystem  error  rate 
(step  9) . 

GENERATING  COMPONENT  FAILURES 

Any  reported  problem  with  the  aircraft  (not  including  technical 
directives,  corrosion  control,  or  planned  maintenance)  results  in  a 
3-M  report.  The  problems  are  called  discrepancies.  Figure  9  shows 
how  reported  discrepancies  result  in  jobs.  Briefly  some  of  the  terms 
can  be  explained  as  follows: 

No  Defect  Squav*  -  if  a  pilot  reports  a  discrepancy  that  cannot  be 
found  by  organizational  maintenance.  A  report  is  filed  indicating 
that  no  defect  could  be  found. 

Repair-In-Place  (RIP)  Job  -  the  discrepancy  is  verified  and  can  be 
corrected  by  organizational  maintenance  without  a  spare  Line  Replace 
(RR)  Job  -able  Unit  (LRU) . 
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Remove/Replace  (RR)  Job  -  (RR)  Job  -  the  discrepancy  is  verified  and 
can  be  repaired  by  organizational  maintenance  by  replacing  with  a 
spare  LRU. 

Cannibalization  Job  -  results  if  a  spare  LRU  is  not  available  after  a 
part  is  removed  (RR  Job).  Cannibalization  jobs  are  only  generated 
following  an  RR  Job  and  are  not  included  in  the  model  at  this  time. 

Troubleshoot  Job  -  results  when  the  troubleshoot  function  requires 
more  than  a  few  minutes.  When  the  actual  problem  is  isolated,  a  RIP 
or  RR  Job  is  generated  to  correct  it.  If  troubleshooting  is 
accomplished  within  a  time  considered  "reasonable"  by  the  technician, 
the  troubleshooting  time  is  included  with  the  RIP  or  RR  Job. 

The  3-M  statistics  generate  all  jobs  from  reported  discrepancies.  The 
AMES  Model  divides  reported  discrepancies  into  three  categories. 

o  False  Discrepancies  -  those  discrepancies  where  no  actual  defects 
were  present.  No  Defect  Squawks  and  unjustified  removals  (Type  I 
error)  are  "False"  Discrepancies. 

o  Maintenance-Caused  Discrepancies  -  those  discrepancies  that  remain 
on  the  aircraft  after  maintenance  to  correct  the  problem  has  been 
completed.  Undiagnosed  failure  (Type  II)  and  damage  (Type  d)  errors 
are  caused  by  faulty  maintenance. 

o  Actual  Discrepancies  -  those  of  the  reported  discrepancies  that  do 
not  fall  in  the  above  two  categories.  Actual  component  defects 
requiring  report  cause  these  discrepancies. 

The  AMES  Model  job  generation  technique  is  shown  in  Figure  10.  It  is 
especially  important  to  note  that  maintenance-caused  deficiencies 
result  from  other  jobs.  Reducing  these  deficiencies  by  reducing 
errors,  reduces  the  total  number  of  jobs.  The  reduced  number  of  jobs, 
in  tum,  reduces  the  number  of  errors  again.  Figure  10  represents  the 
true  situation  more  accurately.  Even  if  an  aircraft  that  almost  never 
failed  could  be  built,  there  would  still  be  discrepancies  due  to  human 
error. 
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Figure  10.  AMES  Model  Corrective  Maintenance  Job  Generation  Overview 
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SECTION  VI 

TOE  AMES  MODEL 


In  light  of  the  previous  discussion,  the  AMES  Model  can  now  be 
described  in  terms  of  the  general  model  framework.  That  is,  the  AMES 
Model  can  be  summarized  in  terms  of  the  following  fundamental  parts: 

•  A  definition  of  the  objectives  of  AMES 

•  An  explanation  of  the  scope  of  the  model 

•  A  functional  description  of  its  operation 

•  An  outline  of  the  input  required  by  AMES 

•  An  outline  of  the  output  produced  by  AMES 

OBJECTIVES 

AMES  has  been  developed  to  study  the  relationship  between  maintenance 
and  system  performance.  More  specifically,  AMES  helps  show  how  the 
effectiveness  of  maintenance  personnel  affects  the  system  performance 
of  an  aircraft  squadron.  To  do  this,  it  is  necessary  first  to  measure 
maintenance  effectiveness  in  quantitative  terms.  This  has  been 
accomplished  in  AMES  by  treating  maintenance  effectiveness  in  terms  of 
performance  times  and  error  rates,  as  described  in  Section  V  and 
Appendix  A. 

Productivity  is  defined  in  terms  of  aircraft  operational  readiness  and 
mission  completion  rate.  With  productivity  so  defined,  AMES  can 
analyze  the  sensitivity  of  productivity  to  maintenance  effectiveness. 
This  is  done  by  varying  error  rates  and  performance  times  (maintenance 
effectiveness)  and  observing  the  resulting  effect  on  aircraft 
operational  readiness  and  mission  completion  rate  (system 
performance).  Thus,  the  AMES  Model  can  help  to  identify  those  levels 
of  maintenance  effectiveness  which  optimize  aircraft  operational 
readiness  within  feasibility  constraints  of  personnel  and  other 
related  maintenance  costs. 

SCOPE 

AMES  is  a  model  of  the  maintenance  and  operation  of  a  carrier-based 
squadron  of  naval  aircraft.  AMES  is  a  dynamic  representation  of  a 
real  system  and  can  be  divided  into  several  segments,  each  of  which 
represents  a  corresponding  segment  of  the  real  system.  The  detail 
with  which  a  given  segment  is  represented  depends  upon  the  interest  of 
that  segment  to  the  current  objectives  of  WES. 
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Areas  Not  Currently  of  Interest  to  AMES 

Those  segments  of  the  system  which  have  little  relationship  to  the 
factors  being  studied  have  been  simplified.  They  are  represented  in 
only  enough  detail  to  complete  the  model.  An  aircraft  undergoing  a 
mission,  for  example,  is  modeled  merely  by  placing  the  aircraft  in  a 
particular  set.  All  details  of  the  mission  other  than  times  of 
takeoff  or  landing  are  ignored.  This  simplication  is  made  because  the 
details  of  a  mission  have  no  effect  on  the  factors  being  studied  by 
AMES. 

Areas  of  Interest  to  AMES 


Those  segments  which  are  directly  related  to  the  objectives  of  AMES 
are  modeled  in  adequate  detail  to  perform  the  desired  analyses.  Some 
of  the  more  important  segnents  include  the  aircraft,  the  components, 
human  performance,  planned  maintenance,  corrective  maintenance, 
component  failures,  and  errors.  Exactly  how  these  and  other 
subsystems  of  AMES  are  represented  is  described  in  detail  in  Appendix 
A. 

Limitations 


Since  AMES  is  a  model  of  an  aircraft  squadron  and  not  the  actual 
squadron  itself,  AMES  will  not  behave  in  exactly  the  same  way  as  the 
real  aircraft  squadron.  Since  AMES  is  a  simplification,  it  cannot 
respond  to  many  of  the  stimuli  that  might  affect  a  real  aircraft 
squadron,  such  as  hostile  enemy  activity  or  fuel  shortages.  Some  of 
the  peculiarities  of  a  carrier-based  naval  squadron  are  built  into 
AMES.  To  use  AMES  to  model  other  similar  systems,  such  as  a 
ground-based  aircraft  squadron,  modifications  are  required.  However, 
AMES  has  been  designed  to  simplify  such  modifications. 

FUNCTIONAL  DESCRIPTION 

A  detailed  functional  description  of  the  AMES  model  can  be  found  in 
Appendix  A.  For  the  purpose  of  this  discussion,  the  function  will  be 
sunmarized  briefly. 

AMES  can  be  considered  as  three  basic  components  and  the  interactions 
between  them.  The  components  are:  a  supply  of  aircraft,  a  schedule 
of  missions,  and  a  maintenance  facility.  There  are  three  possible 
interactions  between  these  three  components.  Two  of  these  are 
manifest  in  the  AMES  Model:  The  interaction  between  missions  and 
aircraft  and  the  interaction  between  aircraft  and  maintenance. 
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The  interaction  between  missions  and  aircraft  is  a  simple  one. 
Whenever  a  mission  is  scheduled  and  two  operationally  ready  aircraft 
are  available,  the  mission  is  flown.  Whenever  a  mission  is  scheduled 
and  two  operationally  ready  aircraft  are  not  available,  the  mission  is 
deferred  for  a  period  (called  the  scrub  time)  or  until  operationally 
ready  aircraft  become  available.  If  the  mission  is  not  flown  before 
the  end  of  the  scrub  time,  the  mission  is  cancelled  (scrubbed). 

The  interaction  between  the  aircraft  and  the  maintenance  facility 
involves  planned  (or  preventive)  maintenance  and  corrective 
maintenance.  In  one  mode,  the  maintenance  facility  attempts  to 
perform  planned  maintenance  actions  on  aircraft  which  are  not  on 
flight  duty.  These  actions  are  performed  as  much  as  possible  in 
accordance  with  a  planned  maintenance  schedule.  In  the  other  mode, 
the  maintenance  facility  attempts  to  correct  aircraft  failures  as  they 
occur.  Both  modes  of  maintenance  operate  simultaneously  and  comprise 
the  interaction  between  maintenance  and  aircraft. 

The  objective  underlying  the  interaction  between  aircraft  and  missions 
is  to  fly  as  many  of  the  scheduled  missions  as  possible.  The 
objective  underlying  the  aircraft-maintenance  interaction  is  to 
continually  maintain  the  highest  level  of  overall  squadron  operational 
readiness. 

INPUT 

The  AMES  Model  is  a  representation  of  an  actual  aircraft  squadron. 
The  framework  of  this  representation  is  based  upon  three  abstract 
notions:  Entities,  attributes,  and  sets.  An  entity  is  an  item  or 

"thing",  such  as  an  aircraft.  An  attribute  is  a  characteristic  of  one 
of  the  entities,  e.g.,  the  number  of  hours  an  aircraft  has  been  flown. 
A  set  is  a  collection  of  entities,  such  as  the  set  of  operationally 
ready  aircraft. 

The  structure  of  any  system  can  be  described  with  these  basic 
abstractions.  The  resulting  representation  will  have  sufficient 
generality  to  model  any  particular  instance  of  that  system.  This 
generality,  however,  creates  a  requirement:  the  user  must  describe 
the  particular  instance  (of  the  system)  that  he  wishes  to  study.  This 
is  done  by  supplying  a  database  which  "fits"  the  structure  of  the 
system.  This  database  determines  the  characteristics  of  an  instance 
of  the  general  system. 
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AMES  Database 


As  mentioned  above,  the  AMES  database  must  fit  the  structure  of  the 
AMES  Model.  This  structure,  or  data  structure,  is  described  in 
complete  detail  in  the  AMES  Source  Code,  and  in  slightly  lesser  detail 
in  the  AMES  Design  Document.  The  database  itself  is  expected  by  the 
AMES  Model  to  follow  a  certain  format.  The  more  important  aspects  of 
that  format  are  described  in  Section  V. 

Program  Control  Parameters 

In  addition  to  the  database  just  mentioned,  the  AMES  program  expects 
another  set  of  data  which  consists  of  parameters  that  control  the 
operation  of  the  program.  These  include  such  parameters  as  the 
duration  of  the  run,  the  frequency  of  each  report,  etc. 

OUTPUT 

The  AMES  program  generates  two  kinds  of  output:  reports  and  traces. 
Both  forms  are  controlled  by  user  input. 

Reports 

Each  report  consists  of  certain  information  which  describes  a 
particular  aspect  of  the  current  condition  of  the  simulated  squadron. 
The  "Status  Report",  for  example,  indicates  the  status  of  each 
aircraft  at  the  time  of  the  report.  The  frequency  of  each  report  must 
be  specified  by  the  user  as  input.  A  detailed  description  of  the 
reports  may  be  found  in  Section  IV. 

Traces 


Traces  were  originally  built  into  the  AMES  program  as  a  debugging 
tool.  When  an  error  occurred  during  the  development  of  AMES,  traces 
facilitated  the  location  or  source  of  the  error.  Although  this 
function  is  currently  unnecessary,  the  traces  were  left  in  the  program 
for  two  reasons.  First,  they  can  still  be  used  for  debugging  in  case 
of  future  developmental  work.  Secondly,  the  traces  can  be  used  to 
follow  the  execution  of  the  program  in  greater  detail.  Peculiarities 
and  uncertainties  can  be  more  closely  examined  by  "tracing"  that  part 
of  the  run  in  which  they  occur.  Traces  are  turned  on  or  off  by  user 
input. 
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SECTION  VII 
RESULTS 


This  section  describes  results  of  the  AMES  project.  A  functional 
model  (A'lES)  has  been  produced.  We  obtained  three  kinds  of  results: 
one  from  the  3-M  data  analysis  required  by  AMES,  another  from 
application  of  the  model,  and  the  last  is  the  model  itself. 

RESULTS  FROM  3-M  ANALYSIS 

The  model  requires  quantitative  input  data,  primarily  to  establish  a 
realistic  baseline  for  future  model  studies.  The  results  are  valuable 
in  the  absence  of  AMES  because  they  give  a  quantitative  picture  of 
present  maintenance.  The  F-14A  was  chosen  to  test  the  model.  We 
present  data  here  that  relates  either  to  the  entire  F-14A  aircraft  or 
to  one  of  the  five  subsystems  that  require  the  most  maintenance.  All 
of  the  data  is  taken  from  a  sampling  of  six  F-14A  squadrons.  Records 
for  each  squadron  extend  for  one  year. 

Number  of  Aircraft 

Using  the  3-M  statistics,  we  can  develop  a  "picture"  of  the 
average  squadron.  It  has  12  aircraft.  Of  these,  57.6  percent 
(6.92  aircraft)  were  operationally  ready  (OR)  at  any  given 
time.  The  remaining  42.4  percent  (5.08  aircraft)  were  not 
operationally  ready  (NOR) .  A  more  detailed  breakdown  of  these 
figures  is  shown  in  Table  7. 

Maintenance  Manhours  Per  Flight  Hour 

For  each  hour  of  flight,  an  F-14A  requires  43.2  manhours  of 
organizational  level  maintenance.  This  total  includes  all  planned 
maintenance  (PM)  and  corrective  maintenance  (CM).  Also  included  are 
all  support  actions  (SAF)  and  technical  directive  compliance  (TDC)  . 
Table  8  shows  the  breakdown  of  hours  into  those  categories. 

Errors 


Using  the  error  analysis  described  earlier,  one  can  conservatively 
determine  actual  error  rates  by  component.  The  number  of  corrective 
maintenance  errors  represents  20  percent  of  the  corrective 
maintenance  actions  (CMAS) . 
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TABLE  7.  AVERAGE  SQUADRON  OF  F-14A  AIRCRAFT 
COMPILED  FROM  3-M  STATISTICS 


AIRCRAFT 

DETAILED 

NUMBER 

CONDITION 

CONDITION  CODE 

OF  AIRCRAFT 

OR 

RMCM  -  U 

.38 

NFE 

.39 

NSC 

6.20 

TOTAL  OR 

6.92 

NOR 

NORM  -  S 

.48 

NORM  •  U 

2.00 

NORS 

2.61 

TOTAL  NOR 

5.08 

TOTAL  OR  AND  NOR  12.00 


Note:  The  total  number  was  chosen  to  be  the  nominal  squadron  size,  12  aircraft. 


LEGEND: 


OR 

NOR 

FSC 

NFE 

RMCM  •  U 
NORM  ■  S 
NORM  ■  U 


Operationally  Ready 
Not  Operationally  Ready 
Full  Systems  Capable 
Not  Fully  Equipped 

Reduced  Material  Condition  due  to  Unscheduled  Maintenance 
NOR  due  to  Scheduled  Maintenance 
NOR  due  to  Unscheduled  Maintenance 
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TABLE  8.  F-14A  ORGANIZATIONAL  MAINTENANCE  MANHOURS 

PER  PLIGHT  HOUR 

MAINTENANCE  CATEGORY  MANHOURS/FLIGHT  HOUR 

PLANNEO  MAINTENANCE  (PM)  19.2 

CORRECTIVE  MAINTENANCE  (CM)  16.4 

SUPPORT  ACTIONS  (SAF)  6.4 

TECHNICAL  DIRECTIVE  COMPLIANCE  (TDC)  1.2 

TOTAL  43.2 
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For  some  subsystems,  Type  I  errors  are  most  prevalent;  for  others. 

Type  Il/d  errors  are  most  common*  Table  9  shows  the  error  rate  in 
percentages  for  the  five  subsystems  that  represent  the  most  work.  The 
figure  also  shows  the  percentage  of  the  CMAS  where  the  parts  from 
supply  are  bad.  Table  10  shows  the  error  rates  grouped  by  work 
center . 

The  discussion  of  error  rates  in  a  previous  section  mentioned  that  the 
Type  I  and  Type  Il/d  error  rates  are  conservative.  The  actual  error 
rates  are  probably  considerably  higher  than  those  used  in  the  model. 

Frequency  of  Failures 

The  Mean  Time  Between  Corrective  Maintenance  Action  (MTBCMA)  is  the 
average  duration  of  flight  between  Corrective  Maintenance  Actions 
(CMAs) .  If  defects  caused  by  maintenance  errors  are  distinguished 
from  actual  hardware  defects,  we  can  identify  two  important  items. 

The  traditional  MTBCMA  is  calculated  based  on  the  total  number  of  CMAs 
(not  including  cannibalization).  The  Mean  Time  Between  Actual 
(hardware)  Defects  (MTBAD)  is  based  on  defects  where  no  error  has 
occurred.  To  determine  a  MTBAD  in  the  absence  of  human  error  we  use 
the  total  CMAs  not  resulting  from  error.  Table  11  shows  both  rates 
for  the  five  subsystems  with  the  most  CMAs.  The  percentage  of  CMA  and 
CM  manhours  represented  by  the  subsystem  is  also  shown  in  the  figures. 

Parts  Delays 

Figure  11  shows  the  types  of  CMA.  The  figure  shows  percentages  for 
cannibalization  parts  required  and  delays.  Of  all  CMAs,  18.3 
percent  require  parts  delays  greater  than  5  days.  The  average 
of  these  delays  is  38  days .  '  ‘ 

RESULTS  FROM  APPLICATION  OF  THE  MODEL 

The  model  produces  two  types  of  results:  understanding  and 
quantitative  results.  As  understanding  of  the  model  and  the 
maintenance  process  increases,  trust  in  quantitative  results  should 
also  increase.  In  this  subsection,  two  examples  of  new  understanding 
and  one  quantitative  result  will  be  discussed.  These  are  preliminary 
results.  They  are  used  primarily  for  illustration. 

Planned  Maintenance  Manhours 


In  our  initial  use  of  the  model,  work  times  for  PM  were  chosen  from 
"book  times",  i.e.,  specifications.  In  this  case,  these  values  for 
various  PM  job  segments  were  taken  from  Maintenance  Requirements  Cards 
(*KC)  for  the  F-14A  Aircraft. 
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Total  Occurences  of  Bad  Parts 
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TABLE  10.  SUMMARY  ~  CORRECTIVE  MAINTENANCE  ERRORS 


WORK  CENTER 

TYPE  1* 

TYPE  ll/d** 

ALL  ERRORS 

Organizational  Maintenance 
Department 

14.5 

9J3 

13.7 

Power  Plants  (110) 

5.9 

15.9 

17.0 

Airframe  (120) 

6.8 

16.0 

17.3 

Corrosion  Control  (121) 

0 

6.4 

6.4 

Aviator  Equipment  (131) 

0 

4.1 

4.1 

Safety  Equipment  (132) 

17.8 

11.0 

14.9 

Electronics  (210) 

13.0 

8.5 

12.1 

Electrical  Instruments  (220) 

17.7 

6.7 

10.4 

Armaments  (230) 

5.4 

6.6 

7.6 

Electro- Weapons  Control  (232) 

15.8 

7.3 

14.7 

Troubleshooters  (320) 

0 

12.5 

12.5 

‘Expressed  as  percent  ol 

'  RR  jobs. 

''Expressed  as  percent  of  all  CM  actions. 

All  CM  actions  is  the 

sum 

TS  +  RIP  +  RR  +  CANN 

where:  TS 

Total 

Troubleshoot  Jobs, 

RIP 

Total 

Repair  In  Place  Jobs, 

RR 

Total 

Remove  and  Replace  Jobs,  and 

CANN  = 

Total 

Cannibalization  Jobs 

All  Errors  is  the  sum 

El  +  EIID  +  BP 

where:  El  = 

Total 

Errors  of  Type  1, 

EIID 

Total 

Errors  of  Type  II  or  d,  and 

BP 

Total 

Occurrences  of  Bad  Parts 
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TABLE  11.  FREQUENCY  OF  CORRECTIVE  MAINTENANCE - 
TOP  FIVE  SUBSYSTEMS 


ALL  CORRECTIVE 
MAINTENANCE 
ACTIONS  (100%) 


NOTE:  18.3%  qf  all  Corrective  Maintenance  Actions 
entail  a  delay  for  parts  greater  than  five  dat 
These  extended  delays  average  38  days. 


NAVTRAEQUIPCEN  77-D-0028-1 


The  number  of  maintenance  manhours  suggested  by  the  book  times 
differed  substantially  from  the  results  of  the  model  run.  The  values 
for  the  3-M  analysis  show  an  average  of  19.6  planned  maintenance 
manhours  per  flight  hour.  Initial  model  results  yielded  about  nine  PM 
manhours  per  flight  hour.  A  sixty  percent  increase  over  book  times 
increased  the  model  results  to  about  18  PM  manhours  per  flight  hour. 

This  indicates  that  book  time  is  substantially  below  actual  time. 
Further  discussion  with  and  observation  of  maintenance  personnel  are 
required  to  improve  the  model  and  discover  how  realistic  the  book 
values  are. 

Operational  Readiness  and  Missions  Completed 

In  3-M  statistics.  Operational  Readiness  (OR)  is  traditionally  taken 
as  the  primary  indicator  of  the  quality  of  system  (and  therefore 
maintenance)  performance.  Initial  runs  indicate  that  when  a 
substantial  percentage  of  flights  cannot  be  flown,  mission  completion 
is  an  equally  important  measure  of  performance.  Considering  only  the 
operational  readiness  parameter  could  be  quite  misleading. 

A  squadron  with  no  missions  for  a  month  could  have  a  100 
percent  OR  status.  As  missions  are  flown,  failures  occur. 
These  failures  require  maintenance  which  subtract  from  the 
OR  status. 

When  the  missions  demands  are  high  and  error  rates  are  reduced,  more 
missions  are  completed.  The  additional  flight  hours  of  these 
additional  missions  result  in  more  failures.  The  additional  failures 
could  even  reduce  operational  readiness  below  the  base  line  level 
(prior  bo  error  reduction) . 

This  means  that  mission  completion  increases  more  dramatically  than 
operational  readiness  when  error  rates  are  reduced.  We  have  found 
this  to  be  true  in  the  initial  two  year  model  runs. 

Figure  12  shows  results  from  four  two-year  runs.  Each  run  is 
indicated  by  a  dot.  A  line  is  used  to  connect  the  dots.  The  figure 
shows  that  a  90  percent  decrease  in  the  error  rate  causes  the 
percentage  of  missions  completed  to  increase  from  49.8  percent 
to  57.1  percent.  To  achieve  the  increase  without  a  reduction 
in  error  rate  would  require  the  addition  of  least  one  entire 
aircraft  to  the  squadron.  Thus,  for  the  particular  parameters 
of  these  runs ,  a  90  percent  reductionin  error  rates  is  approx¬ 
imately  equivalent  to  adding  another  aircraft  to  the  squadron. 

The  model  is  still  very  new  and  has  not  been  thoroughly  validated. 
Thus,  the  above  results  should  be  treated  as  an  indicator  only.  Both 
the  model  and  maintenance  as  a  system  need  to  be  studied  and 
understood  better  before  specific  relationships  can  be  accepted. 
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%  REDUCTION  IN  MAINTENANCE  ERRORS 


Figure  12.  Error  Reduction  for  Each  of  Four  Two-Year  Simulations 
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THE  MODEL  ITSELF 

The  model  itself  is  a  result.  At  the  completion  of  this  study,  only 
the  first  step  has  been  made  toward  a  better  understanding  of  the 
interaction  of  human  performance  with  the  maintenance  process.  The 
model  is  at  present  a  strong,  flexible  model  that  simulates 
maintenance.  It  has  been  designed  and  executed  so  that  it  is  easy  to 
understand.  In  the  next  step  of  modifying  the  model  to  increase  the 
verisimilitude  of  the  simulation,  we  will  learn  a  great  deal  more 
about  maintenance. 
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SECTION  VIII 

CONCLUSIONS  AM)  RECOMMENDATIONS 


The  AMES  project  was  intended  as  a  step  toward  establishing  a 
quantitative  relationship  between  the  effectiveness  of  maintenance  and 
the  operational  readiness/mission  completion  of  aircraft  served  by 
maintenance  personnel.  The  quantitative  relationship  should  indicate 
the  tremendous  importance  of  people-related  efforts  alongside  of 
hardware  development. 

In  undertaking  AMES,  we  have  two  long  range  goals  for  the  application 
of  the  AMES  results: 

•  To  provide  a  tool  enabling  R&D  managers  to  assign  higher 
priorities  to  people- related  efforts  in  maintenance 
research. 

•  To  provide  a  basis  for  training  new  maintenance  managers  to 
understand  maintenance  as  a  system. 

Our  iirmediate  objective  on  the  road  to  these  two  goals  can  be  stated 
as  follows: 

•  To  create  a  prototype  functional  simulation  model  for 
studying  aircraft  maintenance  effectiveness. 

The  manner  in  vhich  these  goals  and  this  objective  have  been  satisfied 
or  advanced  is  discussed  in  the  first  part  of  this  section.  In  the 
second  part,  recommendations  are  presented.  The  third  part  consists 
of  a  closing  statement. 

CONCLUSIONS 

The  accomplishments  of  the  project  are  described  under  the  following 
headings: 

•  AMES  -  A  Prototype  Functional  Simulation  Model  for  Aircraft 
Maintenance  Effectiveness 

•  AMES  -  A  Tool  for  R&D  Managers 

•  AMES  -  A  Basis  for  Training  Maintenance  Managers 
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AMES  -  A  Prototype  Functional  Simulation  Model  for  Aircraft 
Maintenance  Effectiveness 


The  flying  and  maintenance  of  aircraft  comprise  a  system  of  extreme 
complexity.  Because  of  the  complicated  way  in  which  the  many  elements 
of  the  system  interact,  the  changes  in  system  performance  of  one 
element  are  difficult  to  predict.  For  instance,  it  is  difficult  to 
predict  quantitatively  how  the  number  of  missions  completed  might 
change  when  maintenance  error  rates  are  reduced.  A  functional 
simulation  model  overcomes  these  difficulties. 

A  functional  simulation  model  duplicates  the  functions  of  the  flying 
and  maintenance  process.  Each  of  the  individual  functional  elements, 
such  as  flights  or  corrective  maintenance,  must  be  described 
quantitatively.  The  interactions  between  elements  proceed  in  a  causal 
manner  as  they  do  for  actual  aircraft.  The  use  of  a  computer  to 
accomplish  the  simulation  allows  several  years  of  simulated  aircraft 
flying  and  maintenance  to  occur  in  seconds.  By  counting  how  often  the 
aircraft  flies  or  how  many  hours  an  aircraft  is  operationally  ready 
during  the  simulation,  one  can  discover  the  system  performance.  In 
effect,  our  computer  model  is  a  small,  rapidly  evolving  copy  of  an 
aircraft  flight  and  maintenance  system.  The  functional  model  should 
respond  to  changes  in  resource  allocation,  error  rate,  or  other 
elements  in  a  manner  similar  to  a  real  squadron. 

The  Aircraft  Maintenance  Effectiveness  Simulation  (AMES)  Model  has 
been  designed  and  constructed  to  include  the  following  features: 

•  Hunan  maintenance  performance 

•  Human  maintenance  error  rates 

•  Manpower  levels 

•  Flights  and  failures 

•  Corrective  maintenance  (except  cannibalization) 

•  Turnaround  and  daily  flight  inspections 

•  Planned  maintenance  (except  corrosion  control  and  technical 
directive  compliance) 

•  Parts  and  facilities  delays 

The  model  has  been  programmed  in  a  language  similar  to  English  so  that 
maintenance  personnel  can  be  used  to  help  improve  the  model's 
agreement  with  an  actual  squadron,  lb  duplicate  the  proper  response 
of  the  elements,  we  must  use  statistical  information  from  actual 
squadrons.  Various  data  —  such  as  performance  time  to  complete 
maintenance,  flight  time  between  aircraft  component  failures,  and  rate 
of  human  errors  —  must  be  used  with  the  model  to  provide  a 
quantitative  baseline  of  performance  for  each  of  the  functional 
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elements.  Some  of  the  baseline  data,  such  as  preventive  maintenance 
schedules,  are  available  from  aircraft  documents.  Some,  like  the  rate 
at  which  discrepancies  are  discovered  in  an  aircraft  component,  are 
available  from  Naval  3-M  data.  Other  data,  like  the  maintenance  error 
rates,  were  derived  from  3-M  data  by  our  own  analysis  techniques. 

MES  has  undergone  extensive  testing  during  all  phases  of  construction 
and  at  completion.  The  testing  has  employed  baseline  data  from  actual 
squadrons,  as  well  as  data  chosen  to  exercise  all  features  of  the 
model . 

Squadron  performance  using  AMES  (with  baseline  data  for  the 
F-14A  aircraft)  has  been  represented  for  periods  of  up  to 
two  years  of  simulated  time.  Fluctuation  of  results  from 
these  AMES  computer  runs  indicates  that  reducing  error  rates 
by  90  percent  can  increase  the  mission  completion  rate  by 
15  percent.  (Error  rate  reduction  of  90  percent  by  using 
proper  training  work  aids  is  possible.  These  results,  deter¬ 
mined  by  our  analysis,  are  for  an  error  rate  of  15  percent  - 
one  error  each  six  corrective  maintenance  actions.) 

AMES  -A  Tool  for  R&D  Managers 

A  long-range  goal  is  to  provide  R&D  managers  with  a  tool  enabling  them 
to  assign  higher  priorities  to  people- related  efforts  in  maintenance 
research.  As  an  aircraft  (or  any  weapon  system)  is  designed,  R&D 
managers  must  make  the  important  decisions  determining  design 
trade-offs.  For  instance,  should  the  landing  gear  be  constructed  of 
steel  (which  is  cheaper)  or  titanium  (which  is  lighter  but  more 
expensive)?  These  decisions  are  based  on  cost  effectiveness.  These 
managers  attempt  to  produce  the  most  aircraft  possible  for  the  least 
money.  For  these  decisions  they  require  quantitative  data. 

At  the  present  time  R&D  managers  do  not  include  maintenance 
effectiveness  in  their  set  of  trade-offs.  They  do  not  believe  the 
effects  of  human  error  or  poor  performance  are  important  enough  to 
include  in  their  consideration.  To  show  them  the  importance  of 
maintenance  and  to  allow  them  to  include  maintenance  in  cost 
effectivness  requires  quantitative  data  relating  such  maintenance 
performance  to  system  performance.  System  performance  can  be  related 
to  cost. 

The  initial  and  tentative  AMES  results  given  above  show  that 
reducing  error  rate  by  90  percent  is  equivalent  to  increasing 
performance  by  15  percent.  This  increase  corresponds  to 
between  one  and  two  additional  aircraft  per  squadron  of  12 
aircraft.  At  a  cost  of  $21  million  per  aircraft,  this  repre¬ 
sents  a  $A00  million  saving  on  the  300  aircraft  purchased  so 
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far.  Such  indications  show  the  importance  of  considering  errors  as 
well  as  time- to-per form  when  studying  the  human  factors  of 
maintenance. 

At  a  future  stage  it  is  hoped  that  an  improvement  of  the  AMES  Model 
will  allow  R&D  designers  to  include  human  performance  variables  in 
design  analysis.  Optimizing  the  present  system  may  produce  good 
aircraft,  but  aircraft  that  cannot  be  repaired  cannot  fly.  With  a 
descendent  of  AMES,  cost  effectiveness  can  include  all  aspects  of 
system  performance,  including  people- related  considerations. 

AMES  -  A  Basis  for  Training  Maintenance  Managers 

The  second  long-range  goal  is  to  provide  a  basis  for  training  new 
maintenance  managers  to  understand  maintenance  as  a  system. 

The  maintenance  manager  has  the  job  of  assigning  the  men  and  equipment 
required  to  maintain  aircraft.  Such  a  job  is  very  important, 
especially  considering  the  huge  cost  of  maintenance  errors.  For  the 
F-14A  we  have  found  that  an  aircraft  is  not  available  for  flight  44 
percent  of  the  time. 

Maintenance  managers  have  little  formal  training  to  help  them 
understand  maintenance  as  a  system.  Much  of  the  time,  a  maintenance 
manager  has  received  training  in  only  one  area.  Perhaps  the  best  way 
to  train  a  manager  of  maintenance  operations  would  be  to  give  him  a 
great  deal  of  training  on  the  job.  The  training  would  be  enhanced  by 
allowing  him  to  try  different  strategies  to  begin  to  understand  how 
changes  in  resource  allocation  produce  changes  in  system  performance. 
Such  training  requires  much  time  and  a  squadron  of  aircraft.  This  is 
an  unacceptably  expensive  training  technique. 

There  are  two  ways  that  AMES  can  provide  a  basis  for  training 
maintenance  managers:  as  a  development  tool  for  formulating  new 
maintenance  strategies  and  as  a  training  tool  in  the  form  of  a 
maintenance  simulator. 

The  developmental  purpose  of  AMES  is  to  acquire  a  true  understanding 
of  how  maintenance  works.  The  studies  that  lead  to  a  better 
understanding  will  produce  new  strategies  for  optimal  utilization  of 
maintenance  resources. 

The  training  purpose  of  AMES  is  to  teach  maintenance  managers  to 
improve  effectiveness.  Simulators  are  gaining  acceptance  in 
maintenance  training  because  of  their  comparative  cheapness.  A 
simplified  version  of  the  AMES  Model  is  much  less  expensive  than  using 
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a  squadron  for  training.  Also,  the  results  of  several  months  of 
maintenance  using  a  particular  resource  allocation  plan  could  be  seen 
almost  immediately. 

RECOMMENDATIONS 

In  pursuit  of  a  quantitative  understanding  of  maintenance  as  a  system, 
six  specific  recommendations  are  advanced.  They  are  arranged  in  the 
order  in  which  they  should  be  carried  out. 

1.  Continued  study  and  improvement  of  the  AMES  Model.  The  human 
factors  community  has  always  needed  a  maintenance  system  to  study. 
The  enormous  cost  of  aircraft  has  prevented  a  systematic  study  of  the 
maintenance  system.  The  AMES  Model  provides  such  an  inexpensive  tool. 
AMES  must  be  improved  and  studied  with  help  from  actual  maintenance 
personnel  to  assure  that  AMES  will  more  closely  duplicate  actual 
maintenance  procedures. 

2.  Systematic  sensitivity  studies.  The  maintenance  system  is 
tremendously  complicated.  Changes  in  failure  rates,  human 
performance,  and  mission  schedule  all  interact  to  change  aircraft 
availability.  By  systematically  manipulating  only  a  single  variable 
at  a  time,  interactions  and  trade-offs  can  be  studied  and  understood. 

3 .  Study  of  error  rate  reduction  on  an  actual  subsystem. 

S tudies  should  be  undertaken  to  determine  if  a  90  percent 
reduction  in  error  rate  for  a  single  subsystem  is  feasible. 
Such  results  can  be  extended  to  all  systems  using  the  model 
to  determine  if  such  an  error  reduction  is  cost  effective. 

4.  Train  maintenance  managers  using  AMES.  It  is  not  feasible  to  tie 
up  an  entire  squadron  to  train  those  who  will  make  decisions  on 
maintenance.  WfS  is  an  exact  analog  of  simulation  used  for  training 
pilots  or  maintenance  personnel.  A  simplified  model  of  AMES  could  be 
used  interactively  to  teach  maintenance  managers  their  trade. 

5.  Set  specifications  for  human  performance  relative  to  system 
performance.  Human  performance  in  maintenance  can  be  specified  in 
terms  of  error  rate  and  performance  time.  Hie  AMES  Model  can  be  used 
to  relate  these  specifications  to  system  performance. 

6.  Include  human  maintenance  performance  considerations  in  hardware 
design.  Now  cost  effectiveness  of  hardware  design  trade-offs  can  be 
investigated  during  design.  The  WES  Model  provides  the  possibility 
of  including  the  human  factors  of  maintenance  into  design  decision. 
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CLOSING  STATEMENT 

Modern  analytical  techniques  are  now  used  for  all  phases  of  hardware 
design.  The  people- related  factors  of  maintenance  have  had  almost  no 
analysis  as  they  relate  to  system  performance  variables  of  the 
aircraft.  AMES  is  an  initial  step  toward  indicating  the  importance 
of  these  considerations  and  discovering  strategies  toward  improving 
both  hun an  performance  and  system  performance. 
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APPENDIX  A 

THE  MODEL 


The  Ames  Model  is  best  described  in  the  way  it  was  developed  —  from 
the  "top  down".  In  top-down  design,  the  program  is  described  (and 
implemented)  in  general  levels  first,  and  then  into  levels  of 
increasing  detail. 

The  Ames  Model  is  written  in  the  language  Simscript  II. 5.  One  of  the 
primary  advantages  of  this  language  is  its  readability  by  those  not 
expert  in  computer  work.  The  terms  in  the  following  explanation  are, 
for  the  most  part,  those  actually  used  in  the  program.  The  program 
itself  reads  very  much  like  English.  The  understanding,  gained  at  the 
general  levels  in  the  following  explanation,  may  be  applied  directly 
to  an  understanding  of  the  more  detailed  levels  of  the  program  itself. 

The  model  simulates  the  events  and  activities  of  aircraft  and 
maintenance  personnel  as  time  passes.  It  is  important  to  realize  that 
the  times  in  the  simulation  are  not  equivalent  to  real  time.  For 
example,  three  months  of  simulated  time  for  a  squadron  require  only  a 
few  minutes  of  actual  time  to  be  simulated  by  the  computer.  For  the 
purposes  of  the  following  discussions,  all  times  are  simulated  times. 

The  model  may  be  divided  into  five  levels  of  implementation.  Each 
succeeding  level  is  more  complicated  than  its  predecessor.  At  each 
succeeding  level  more  detail  is  added.  Some  parts  of  the  program  may 
be  changed  substantially  from  level  to  level;  some  parts  not  at  all. 
The  five  levels  of  implementation  are  as  follows: 

I  Flying  the  Aircraft 

II  Hie  Failure  of  Components 

III  Corrective  Maintenance:  Repairing  Failures 

IV  Planned  Maintenance,  Including  Flight  Inspections 

V  Priority  Repair  for  Immediate  Mission  Assignment 

Each  of  the  levels  is  described  with  a  figure.  These  diagrams  attempt 
to  show  how  the  program  is  constructed  at  each  level.  The  diagrams 
actually  follow  the  structure  of  the  program  quite  closely.  The 
following  conventions  are  observed: 

1.  Circles  represent  groups  of  aircraft,  herein  termed  "sets". 

2.  Boxes  represent  procedures  that  act  on  the  aircraft,  either  moving 
them  from  set  to  set  or  working  on  the  aircraft  within  a  set. 
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3.  All  aircraft  must  be  in  one  and  only  one  o£  the  sets,  except  for  a 
brief  time  when  a  procedure  moves  them  from  one  set  to  another. 

4.  The  duty  scheduler  procedure  does  much  of  the  moving  of  the 
aircraft  from  one  set  to  another. 

5.  All  movement  of  aircraft  is  governed  by  an  externally  imposed 
schedule  of  missions.  Every  effort  is  made  to  fly  as  many  missions  as 
possible. 

6.  The  diagrams  are  designed  to  show  movement  and  action  with  respect 
to  the  aircraft.  Other  considerations,  like  temporal  relationships, 
may  be  neglected  for  the  sake  of  clarity. 

7.  A  much  more  detailed  docunent,  written  in  an  English-like  design 
language,  has  been  constructed  to  describe  tne  model  in  much  greater 
detail.  Due  to  its  length  it  has  not  been  included  in  the  report. 
However,  it  is  available  to  facilitate  future  improvement  of  the 
model. 

LEVEL  I  -  FLYING  TOE  AIRCRAFT 

In  the  most  general  sense,  the  purpose  of  aircraft  is  to  fly  missions. 
Each  mission  is  composed  of  one  or  more  sorties.  In  Figure  A-l,  basic 
purpose  is  shown.  The  diagram  depicts  aircraft  operation  if  no 
aircraft  component  ever  failed. 

Aircraft  with  all  systems  operational  are  stored  in  the  FSC  (Full 
Systems  Capable)  POOL.  As  the  simulation  begins,  all  aircraft  are  in 
the  FSC  POOL.  Each  mission  has  a  scheduled  start  time,  a  duration, 
and  a  scrub  time.  The  scrub  time  is  the  delay  allowed  the  aircraft 
before  the  mission  is  cancelled. 

Before  the  missions  of  a  given  day  are  flown,  the  number  of  aircraft 
required  to  carry  out  the  missions  is  determined.  The  desired  number 
of  aircraft  are  moved  from  the  FSC  POOL  to  the  TA  WORKSHOP  (turnaround 
inspection  area) .  Those  aircraft  with  the  fewest  accumulated  flight 
hours  are  selected  first.  Here  they  are  subject  to  inspection  before 
the  first  flight,  after  the  last  flight  and  between  flights.  At  Level 
I  of  the  implementation,  however,  there  are  no  inspections  —  just  a 
place  to  do  them. 

Since  the  aircraft  have  no  failures,  no  planned  maintenance  (PM)  ,  and 
no  inspection,  all  aircraft  are  certified  for  flight  on  passing  out  of 
the  TA  WORKSHOP.  The  aircraft  certified  for  flight  are  put  in  the 
FLIGHT  CERTIFIED  AIRCRAFT  POOL.  They  remain  certified  until  an 
aircraft  is  actually  scheduled  to  fly  or  until  aircraft  are  no  longer 
required. 
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When  the  time  for  a  scheduled  mission  occurs,  BEGINNING  OF  SORTIE 
removes  the  aircraft  from  FLIGHT  CERITIFIED  AIRCRAFT  POOL.  Those 
aircraft  with  the  lowest  number  of  flight  hours  are  selected  first. 
If  they  are  available,  aircraft  are  launched  at  the  scheduled  start 
time.  If  aircraft  become  available  after  the  scheduled  start  time  but 
before  the  scrub  time  has  passed,  they  are  launched  immediately.  If 
aircraft  have  not  become  available  before  the  end  of  scrub  time,  the 
mission  is  cancelled.  After  takeoff,  the  aircraft  is  placed  into  the 
set  of  FLYING  AIRCRAFT.  At  the  launch,  BEGINNING  OF  SORTIE  also 
schedules  an  END  OF  SORTIE  to  complete  the  mission  after  the  scheduled 
duration. 

When  the  duration  of  the  SORTIE  is  completed,  the  END  OF  SORTIE 
occurs.  END  OF  SORTIE  advances  the  flight  time  statistics  of  the 
aircraft.  The  landing  of  the  aircraft,  as  well  as  removing  the 
aircraft  from  FLYING  AIRCRAFT,  occurs  at  END  OF  SORTIE.  Finally,  END 
OF  SORTIE  replaces  the  aircraft  in  TA  WORKSHOP  for  the  turnaround 
inspection  required  after  the  flight. 

As  the  missions  of  the  day  are  flown,  the  computer  keeps  count  of  how 
many  missions  remain.  If  ever  there  are  more  aircraft  in  the  set  of 
FLIGHT  CERTIFIED  AIRCRAFT  POOL  than  are  required  for  all  of  the 
remaining  missions  of  the  day,  the  excess  aircraft  are  removed  by 
REMOVE  EXCESS  FLIGHT  CERTIFIED  AIRCRAFT.  The  removed  aircraft  are 
placed  in  the  FSC  POOL.  The  aircraft  are  removed  from  FLIGHT 
CERTIFIED  AIRCRAFT  POOL  based  on  the  total  number  of  flight  hours: 
those  aircraft  with  the  most  flight  hours  are  removed  first. 

LEVEL  II  -  THE  FAILURE  OF  COMPONENTS 

The  simple  picture,  diagrammed  in  Figure  A-l,  describes  aircraft 
operations  in  the  absence  of  failures.  Level  II  of  the  model  includes 
component  failures.  The  aircraft  picture  is  enlarged  to  include 
components.  For  the  purposes  of  this  level  of  the  model,  the  aircraft 
is  a  flying  set  of  components  with  flight  statistics.  From  the  data 
analysis,  we  know  the  Mean  Time  Between  Failures  (MTBF)  for  each  of 
the  subsystems  of  the  aircraft.  AMES  is  designed  so  that  although 
specific  component  failures  occur  randomly  for  a  given  subsystem,  the 
average  time  between  failures  frcm  the  model  run  will  approximate  the 
actual  MTBF. 

Each  of  the  simulated  aircraft  then  contains  a  list  of  all  components. 
For  each  component  the  list  includes  the  following  information: 
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•  Whether  the  component  is  good,  defective  or  removed. 

•  The  criticality  of  the  failure  (whether  the  next 
failure  renders  the  aircraft  NOR) . 

•  The  specific  number  of  flight  hours  until  the  next 
failure  for  a  particular  component. 

The  diagram  of  Figure  A-l  may  be  extended  to  include  these 
modifications.  Figure  A-2  shows  the  diagram  including  component 
failures.  This  second  level  of  detail  includes  all  items  of  the 
previous  level,  but  to  accommodate  the  component  failures,  the 
following  modifications  have  been  added. 

1.  TA  WORKSHOP  -  a  turnaround  to  check  for  defective  components. 

2.  DUTY  SCHEDULER  -  the  decision  whether  a  particular  aircraft  may  be 
scheduled  for  flight. 

3.  OR  MAINTENANCE  POOL  -  the  set  of  OR  aircraft  to  be  repaired. 

4.  NOR  MAINTENANCE  POOL  -  the  set  of  NOR  aircraft  to  be  repaired. 

5.  INITIATE  CM  (Corrective  Maintenance)  to  remove  aircraft  that  fail 
to  pass  inspection  to  the  NOR  MAINTENANCE  POOL. 

Before  the  simulation  begins,  the  flight  time  mtil  the  next  failure 
for  each  component  of  each  aircraft  is  determined  from  actual  flight 
failure  statistics,  as  is  the  criticality  of  the  failure.  At  this 
stage  of  implementation,  repair  jobs  are  completed  instantaneously. 

The  simulation  begins  again  with  mission  requirements  for  aircraft. 
The  DUTY  SCHEDULER  attempts  to  find  enough  aircraft  to  fly  the 
mission.  Aircraft  are  selected  for  flight  in  the  following  order  of 
preference. 

1.  Aircraft  with  all  components  good  (FSC  POOL). 

2.  Operationally  Ready  aircraft  that  require  maintenance  but  are  not 
already  flying  (these  are  found  in  OR  MAINTENANCE  POOL) . 

3.  Those  aircraft  that  have  failures  that  render  them  not 
operationally  ready  (NOR)  are  kept  in  NOR  POOL.  These  aircraft  may 
not  be  flown  until  all  serious  defects  are  corrected. 
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The  aircraft  designated  for  flight  by  the  DUTY  SCHEDULER  are  moved 
into  the  TA  WORKSHOP  to  undergo  the  turnaround  inspection.  If  more 
aircraft  are  required  than  can  be  found,  the  Duty  Scheduler  waits 
until  repair  is  completed  on  an  NOR  aircraft.  When  the  repair  has 
been  completed,  the  aircraft  becomes  OR  and  is  moved  to  OR  MAINTENANCE 
POOL  from  the  NOR  MAINTENANCE  POOL.  Should  enough  aircraft  to  fly  a 
mission  not  be  available  at  the  time  of  the  scheduled  takeoff,  the 
mission  is  deferred  in  BEGINNING  OF  SORTIE.  Normally  two  aircraft  are 
required  for  a  mission.  If  the  aircraft  are  still  unavailable  after 
the  scrub  time,  the  mission  is  cancelled. 

The  inspections  consist  of  checking  each  component  of  a  particular 
aircraft  to  see  if  it  has  failed  since  the  last  inspection.  Aircraft 
with  components  that  fail  but  do  not  render  the  aircraft  NOR  may 
continue  to  fly.  A  single  NOR  component  failure  results  in  the 
aircraft  being  classified  NOR.  No  further  flights  for  this  aircraft 
may  occur  until  its  NOR  components  are  corrected.  Components  on 
aircraft  that  are  still  OR  but  continue  to  fly  may  fail  again  and 
become  NOR. 

LEVEL  III  -  CORRECTIVE  MAINTENANCE:  REPAIRING  FAILURES 

The  third  level  actually  repairs  the  aircraft.  At  this  level,  we  must 
include  the  following: 

•  Different  Types  of  Repair  Jobs 

•  Personnel 

•  Details  of  the  Various  Jobs 

•  Errors 

Since  the  purpose  of  AMES  is  to  relate  the  system  performance  to 
various  characteristics,  the  third  level  contains  many  of  the 
important  details.  Each  of  the  above  topics  will  be  treated  in  some 
detail. 

Different  Types  of  Repair  Jobs 

The  basic  structure  shown  in  Figure  A- 2  has  been  expanded  to  include 
practically  all  of  corrective  maintenance  (CM)  in  Figure  A-3.  The 
third  level  implements  the  details  of  the  CM  JOBS.  The  following  are 
the  5  types  of  CM  JOBS: 

REPAIR-IN-PLACE  (RIP)  JOB  -  the  defective  component  can  be  repaired  in 
place  on  the  aircraft.  Parts  may  be  needed. 

REMOVE/ RE PLACE  (RR)  JOB  -  the  defective  component  is  removed  and 
replaced  with  another.  RR  JOBS  may  lead  to  cannibalizing  other 
aircraft  to  speed  repair. 
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TROUBLESHOOTING  (TS)  JOB  -  If  the  troubleshooting  time  extends  beyond 
the  normal  5  to  10  minutes,  the  troubleshooting  action  is  considered 
to  be  a  separate  job.  Following  TS  JOB  an  RR  or  RIP  JOB  is  generated 
to  repair  the  component. 

NO  DEFECT  (ND)  JOB  -  the  troubleshooting  job  fails  to  detect  the 
failure  reported  by  the  flight  crew  (squawk). 

CANNIBALIZATION  (C)  JOB  -  the  parts  required  for  an  RR  JOB  have  long 
delays.  The  good  part  is  removed  from  a  NOR  aircraft  to  hasten 
repair.  When  the  part  finally  does  arrive,  it  is  replaced  into  the 
NOR  aircraft  that  was  cannibalized.  Cannibalization  has  been  designed 
but  not  yet  included  in  the  model. 

The  CM  jobs,  with  the  exception  of  cannibalization,  are  shown 
schematically  in  Figures  A-4,  A-5,  A-6  and  A-7.  All  of  these  jobs 
occur  when  the  aircraft  are  in  the  OR  MAINTENANCE  POOL  or  the  NOR 
MAINTENANCE  POOL.  The  principal  distinguishing  features  are  the 
following: 

REPAIR-IN-PLACE  (RIP)  JOB  (Figure  A-4).  The  job  begins  when  an 
aircraft  is  placed  in  the  OR  MAINTENANCE  POOL  or  the  NOR  MAINTENANCE 
POOL  after  a  component  failure  has  been  detected.  People  are  required 
briefly  to  determine  if  facilities  are  available  and/or  parts  are 
required.  If  either  is  required,  the  program  waits  until  both  are 
available.  The  time  of  the  wait  is  determined  statistically.  This 
will  be  explained  later. 

Once  parts  and  facilities  are  available,  the  job  commences  as  soon  as 
the  proper  complement  of  people  is  available.  The  duration  of  the 
work  is  also  determined  statistically.  If  a  Collateral  Duty  Inspector 
(CDI )  has  been  working  on  the  job,  no  inspection  is  required. 
Otherwise  a  CDI  inspection  is  required. 

At  the  completion  of  the  job,  it  is  determined  (based  on  probabilities 
provided  as  input)  whether  an  undiagnosed  failure  (Type  II)  or  damage 
(Type  d)  error  has  been  made.  If  an  error  occurred,  the  component 
remains  failed  but  the  job  is  completed.  The  failed  component  will  be 
found  by  inspection  at  a  later  time.  At  the  time  of  discovery,  a  new 
job  is  generated.  If  a  Type  II  error  does  not  occur,  the  component  is 
reset  to  good . 

REMOVE/REPLACE  (RR)  JOB  (Figure  A-5).  The  RR  job  is  very  similar  to 
the  RIP  job  with  the  following  exceptions: 
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1.  Facilities  are  required  before  it  can  be  determined  if  a  part 
delay  will  be  required. 

2.  Cannibalization  can  occur  to  speed  delivery.  At  this  time, 
cannibalization  is  not  included.  Future  improvements  in  the  model 
will  correct  the  simplification. 

TROUBLESHOOTING  (TS )  JOB  (Figure  A-6).  These  jobs  require  facilities 
only.  Parts  are  not  required.  After  the  facilities  are  available  and 
people  are  found,  work  commences.  At  the  conclusion  of  the  TS  job,  an 
RR  or  RIP  job  is  generated.  Errors  and  CDI  inspection  do  not  occur 
during  this  job. 

NO  DEFECT  (ND)  JOB  (Figure  A-7) .  Following  a  flight  crew  squawk, 
troubleshooting  begins  but  no  defect  is  found.  This  job  is  equivalent 
to  a  TS  Job  where  no  defect  is  found. 

Personnel 


The  model  includes  3  skill  levels  of  personnel: 

Collateral  Duty  Inspectors  (CDIs) 

Senior  Technicians  (SENIORS) 

Junior  Technicians  (JUNIORS) 

They  are  arranged  in  work  centers  in  each  of  several  shifts. 
Initially,  for  each  shift,  a  specified  complement  of  men  is  assigned 
to  a  work  center.  Depending  on  the  subsystem  that  fails,  personnel 
from  the  appropriate  work  center  are  selected  to  repair  the  failure. 
Each  corrective  maintenance  (CM)  job  requires  at  least  1  CDI  or 
SENIOR.  If  no  CDI's  worked  on  the  job,  an  additional  CDI  inspection 
following  completion  of  the  repair  is  required.  Subject  to  the  above 
conditions,  JUNIORS  are  preferentially  chosen  to  work  on  a  job. 

People  are  assigned  preferentially  to  work  on  jobs  with  the  shortest 
estimated  time  to  repair.  At  the  conclusion  of  the  job,  the  people 
are  returned  to  the  work  center  for  reassignment.  At  the  completion 
of  the  shift,  all  jobs  are  stopped,  all  personnel  are  returned  to  the 
work  center.  The  workers  from  the  next  shift  are  then  assigned,  based 
on  the  same  criteria. 

Occasionally,  an  OR  aircraft  is  required  when  none  are  available 
except  those  currently  undergoing  maintenance.  In  such  cases,  the 
DUTY  SCHEDULER  firsts  selects  the  OR  aircraft  with  the  fewest  jobs  to 
be  completed,  then  discontinues  all  jobs  on  the  selected  aircraft  and 
moves  the  aircraft  into  the  TA  WORKSHOP.  When  this  occurs,  the 
personnel  (who  had  been  performing  maintenance  on  the  selected 
aircraft)  are  returned  to  their  respective  work  centers. 
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Figure  A-4.  Repair- In-Place  (RIP)  Jobe 
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Figure  A-5.  Remove/Replace  (RR)  Jobs 
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Figure  A-6.  TraubMwoting  (TS)  Job* 
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Figure  A-7.  No  Defect  (ND)  Jobs 

103/104 


NAVTRAEQUIPCEN  77-D-0028-1 


Much  of  the  information  relevant  to  human  performance  is  derived  from 
statistical  analysis  of  the  actual  data.  This  information  includes 
elapsed  maintenance  time  (EMT)  and  the  number  of  men  required.  For  a 
given  job,  the  EMT  and  number  of  men  are  determined  by  a  random  draw 
from  the  statistical  data. 

Details  of  the  Various  Jobs 


There  are  only  slight  variations  in  details  among  the  CM  jobs,  as 
shown  in  Figures  A-4  through  A-7.  Because  of  basic  similarities  among 
the  jobs,  the  Repair-In-Place  (RIP)  job  will  be  described. 

Once  a  component  failure  is  detected,  the  job  to  complete  the  repair 
is  generated.  The  job  cannot  begin  until  several  conditions  are  met. 
First,  the  aircraft  must  not  be  on  flight  duty,  i.e.,  must  be  in 
either  the  OR  MAINTENANCE  POOL  or  the  NOR  MAINTENANCE  POOL.  Once  an 
aircraft  is  placed  in  the  appropriate  maintenance  pool,  all  repair 
jobs  for  that  aircraft  are  awaiting  personnel.  As  soon  as  people  of 
the  proper  skill  level  become  available  at  the  work  center  that 
repairs  the  defective  component,  the  job  begins.  The  personnel  are 
required  only  for  a  brief  time  to  determine  which  facilities  are 
required.  Should  the  facilities  not  be  available,  the  job  must  wait 
until  they  become  available. 

When  facilities  are  available,  personnel  are  again  required  to 
determine  the  parts  required.  If  the  parts  are  unavailable,  the  job 
can  continue  only  when  the  parts  become  available.  At  this  stage  of 
development,  both  facilities  and  parts  delay  are  determined 
statistically  by  component  from  distributions  derived  from  the  3-M 
data.  Determining  parts  delays  is  simpler  than  determining  facilities 
delays,  but  less  correctly  approximates  reality.  (Should  actual 
quantitative  and  more  correct  measures  of  facilities  and  parts  delays 
be  desired  later,  they  can  be  included  in  the  model.)  All  facilities 
and  parts  required  to  complete  the  job  are  now  present. 

As  people  again  become  available,  they  are  assigned  to  work  on  the 
job.  The  time  required  to  complete  the  job  —  the  elapsed  maintenance 
time  (EMT)  —  is  selected  at  random  from  a  statistical  distribution 
from  the  actual  3-M  data.  After  work,  the  job  is  completed,  and  the 
personnel  are  returned  to  the  work  center.  If  all  personnel  working 
the  job  are  JUNIORS  and  SENIORS,  an  additional  CDI  inspection  is 
required.  A  small  percentage  of  the  EMT  will  have  been  left  over  for 
the  inspection.  When  the  CDI  is  available,  he  is  assigned  to  the  job 
to  complete  the  inspection,  and  is  then  released. 
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The  job  is  now  completed.  The  flight  time  until  the  next  failure 
(FTNF)  is  selected  from  an  observed  distribution.  The  error  check 
which  follows  next  is  described  later  in  this  report.  Following  the 
error  check,  all  statistics  of  the  job  (EMT,  manhours,  parts  delays, 
and  failed  components)  are  recorded. 

The  other  types  of  jobs  contain  substantially  the  same  elements.  The 
troubleshooting  job,  after  it  is  complete,  generates  an  additional  RR 
or  RIP  job  to  do  the  repair  actually  discovered  in  troubleshooting. 

As  soon  as  a  failure  is  detected,  a  job  to  repair  the  component  is 
generated.  The  decision  as  to  which  of  the  types  of  jobs  is  required 
is  determined  from  3-M  statistics  by  component. 

After  all  statistics  have  been  recorded,  the  aircraft  is  returned  to 
the  proper  pool:  OR  MAINTENANCE,  NOR  MAINTENANCE,  or  FSC.  It  has 
been  noted  that  maintenance  can  be  interrupted  on  OR  aircraft  at  any 
time  if  an  aircraft  is  needed  for  flight.  At  the  conclusion  of  the 
required  flights  (when  the  aircraft  is  returned  to  one  of  the 
maintenance  pools)  the  jobs  may  continue  when  facilities,  parts,  or 
people  become  available. 

Errors 


One  of  the  important  aspects  of  the  AMES  Model  is  its  consideration  of 
human  errors.  The  model  includes  the  following  types  of  errors: 

•  Undetected  failure  errors 

•  Damage  errors 

•  Parts  bad  from  supply 

•  Unjustified  removal  errors 

•  No  defect  errors 

The  manner  in  which  each  error  is  incorporated  in  the  model  is  now 
described.  Since  at  this  time  undetected  failure  (Type  II)  errors  and 
damage  (Type  d)  errors  are  difficult  to  separate  in  the  3-M 
statistics,  they  are  considered  collectively  (Type  ll/d) . 

The  manifestation  of  both  undetected  failure  and  damage  errors  is  that 
a  particular  component  on  an  aircraft  is  thought  to  be  good  when  it 
is,  actually  faulty.  The  errors  are  discovered  at  the  completion  of 
all  but  troubleshooting  jobs.  (Following  a  troubleshooting  job,  the 
component  still  has  not  been  repaired.)  When  such  errors  occur,  the 
status  of  the  component  is  "defective".  The  failure  will  be 
discovered  later. 
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Parts  bad  from  supply  are  handled  in  a  similar  way.  If  a  part  is 
defective  vhen  it  is  placed  into  an  aircraft,  it  will  be  detected 
subsequently. 

Unjustified  removal  (Type  I)  errors  cause  removal  of  good  components. 
They  only  occur  in  RR  jobs.  To  handle  this  error,  additional  RR  jobs 
are  generated  statistically. 

No  defect  errors  are  handled  similarly  to  unjustified  removal  errors, 
except  extra  ND  jobs  are  generated. 

LEVEL  IV  -  PLANNED  MAINTENANCE,  INCLUDING  FLIGHT  INSPECTIONS 

The  fourth  level  adds  planned  maintenance  (PM)  (see  Figure  A-8).  This 
level  can  be  divided  into  the  following  sections: 

•  Planned  Maintenance  Overview 

•  PM  Schedul ing 

•  Turnaround  and  Daily  Inspections 

•  PM  Jobs 

Planned  Maintenance  Overview 

Since  each  type  of  PM  is  performed  frequently,  there  are  relatively 
few  problems  like  facility  delays,  parts  delays,  or  human  errors.  PM 
has  been  included  in  a  simple  way  because  it  does  account  for  a 
significant  fraction  of  the  maintenance  manhours.  Occasionally, 
aircraft  availability  is  also  limited  because  of  overdue  PM.  For 
these  reasons,  the  form  of  PM  is  different  from  CM. 

Planned  maintenance,  as  implemented  in  AMES,  is  divided  into  special 
inspections,  phased  inspections,  daily  inspections,  turnaround 
inspections,  conditional  inspections,  and  corrosion  control.  Planned 
replacement  and  compliance  with  technical  directives  are  not  included 
in  the  model  at  this  time. 

All  of  the  above  types  included  in  the  model  have  requirements  based 
on  three  denominations:  calendar  time,  flight  hours,  and  nunber  of 
flights.  Special  inspection  can  be  denominated  in  any  of  the  three. 
Phased  inspection  is  only  denominated  in  flight  hours.  Daily 
inspection  is  only  denominated  in  calendar  time. 

PM  Scheduling 

Each  of  the  PM  and  inspection  requirements  must  be  repeated  at 
specified  intervals  of  calendar  time,  flight  hours,  or  flights.  The 
specified  intervals  will  be  called  the  nominal  intervals.  To 
facilitate  all  other  maintenance,  the  requirements  may  be  accomplished 
at  any  time  within  a  required  tolerance  of  the  nominal  interval. 
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For  example,  a  PM  requirement  every  500  hours  (the  nominal  value)  can 
be  performed  within  a  tolerance  of  10%.  The  requirement  is  within  the 
allowable  limits  if  it  begins  after  at  least  450  flight  hours  and  is 
completed  before  550  flight  hours.  If  the  aircraft  accumulates  in 
excess  of  550  hours  since  the  last  performance  of  the  specified 
maintenance,  it  becomes  NOR  and  may  not  fly  until  the  maintenance  is 
completed.  The  tolerances  are  shown  explicitly  as  follows: 

•  Fbr  requirements  denominated  in  calendar  time,  the  requirements 
must  be  completed  within  3  days  of  the  nominal  value  to  be  within  the 
acceptable  range. 

•  For  requirements  denominated  in  flight  hours  or  flights,  the 
acceptable  tolerance  is  10%  of  the  nominal  value. 

The  program  must  make  every  effort  to  assure  that  people  are  available 
to  accomplish  planned  maintenance  when  it  is  required.  A  system  was 
designed  to  help  the  AMES  Model  effectively  allocate  maintenance 
personnel  to  perform  planned  maintenance  within  acceptable  tolerances. 
The  interval  when  a  planned  maintenance  requirement  may  be 
accomplished  within  tolerance  has  been  divided  into  three  equal 
intervals.  Each  of  the  three  intervals  is  treated  as  a  different 
stage  of  priority,  lb  these,  a  fourth  stage  of  priority  is  added  vhen 
the  requirement  becomes  overdue.  The  four  stages  of  priority  are  as 
follows : 

1.  Early  -  first  interval  -  personnel  will  be  assigned,  if  available, 
if  the  aircraft  is  in  the  OR  MAINTENANCE  POOL  or  the  NOR  MAINTENANCE 
POOL.  The  early  stage  is  the  first  one  third  of  the  tolerated 
interval . 

2.  Nominal  -  second  interval  -  personnel  may  be  assigned,  but  only  if 
the  aircraft  is  not  required  for  flight  and  the  maintenance  personnel 
required  are  available. 

3.  Critical  -  third  interval  -  personnel  will  be  assigned  unless  no 
personnel  are  available.  Unless  the  aircraft  is  required  for  flight, 
the  aircraft  will  be  placed  where  maintenance  may  begin  (OR 
MAINTENANCE  POOL  or  NOR  MAINTENANCE  POOL) . 

4.  NOR  -  past  the  tolerated  limits  -  the  aircraft  is  no  longer 
suitable  for  flight  until  the  PM  is  completed.  The  aircaft  will  be 
placed  in  the  NOR  MAINTENANCE  POOL  if  it  is  not  there  already. 
Personnel  will  be  assigned  as  soon  as  they  become  available  with 
preference  given  only  to  inspection  or  maintenance  required  for 
flight. 


NAVTRAEQUIPCEN  77-D-0028-1 
Turnaround  and  Daily  Inspections 

The  turnaround  and  daily  inspections  have  not  been  included  until  this 
level.  Turnaround  and  daily  inspections  are  treated  here  separately 
because  they  are  PM  requirements  that  depend  on  both  flight  and 
calendar  requirements. 

Turnaround  inspection  is  required  before  the  first  flight  of  the  day, 
between  flights,  and  following  the  last  flight.  It  is  not  required 
when  the  aircraft  is  not  flying.  Turnaround  inspection  is  scheduled 
automatically  every  time  the  aircraft  is  put  into  the  TA  WORKSHOP. 
Turnaround  inspection  is  valid  for  24  hours. 

The  daily  inspection  is  required  before  flight.  It  is  valid  for  24 
hours  if  the  aircraft  is  to  fly,  unless  the  aircraft  undergoes  CM.  If 
the  aircraft  does  not  fly,  the  daily  inspection  is  required  every  72 
hours.  At  the  conclusion  of  each  daily  inspection,  a  new  daily 
inspection  is  scheduled  for  72  hours  hence.  Each  time  the  aircraft 
enters  the  TA  WORKSHOP  from  the  FSC  POOL  or  the  OR  MAINTENANCE  POOL, 
it  is  determined  if  the  aircraft  requires  a  daily  inspection  to  fly. 
If  required,  the  daily  inspection  is  done  in  the  TA  WORKSHOP. 

Since  both  turnaround  and  daily  inspections  are  flight  related,  they 
have  the  highest  priority. 

PM  Requirements 

The  structure  of  PM  requirements  differs  from  that  of  CM  jobs.  Each 
PM  requirement  consists  of  a  set  of  segments.  Each  segment  may 
include  work  at  several  work  centers.  That  part  of  the  work  of  a 
segment  at  a  given  work  center  is  called  (by  the  model)  a  PM  work 
center  job  (or  simply  PM  job).  Each  segment  may  have  several  PM  jobs. 
Each  of  these  jobs  has  a  fixed  time.  Figure  A-9  shows  schematically 
how  a  PM  requirement  is  completed. 

Each  of  the  FM  jobs  is  like  a  CM  job,  but  much  simpler.  As  shown  in 
the  insert  of  Figure  A-9,  all  that  is  required  is  to  wait  for  people. 
When  people  are  available,  the  work  commences.  At  completion,  the 
people  are  released  and  statistics  are  recorded. 

When  all  of  the  PM  jobs  in  the  segment  are  completed,  the  next  segment 
is  begun.  After  all  of  the  segments  of  the  PM  requirement  are 
finished,  the  next  PM  requirement  is  scheduled  and  the  aircraft  is 
reassigned. 
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Figure  A-9.  Planned  Maintenance  (PM)  Requirements 
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All  PM  jobs,  except  phased  maintenance,  may  be  interrupted.  Phased 
maintenance  may  not  be  interrupted  until  a  requirement  is  completed. 

LEVEL  V  -  PRIORITY  REPAIR  FOR  INMEDIATE  MISSION  ASSIGNMENT 

Since  the  primary  purpose  of  the  aircraft  is  to  fly  missions,  every 
effort  must  be  made  to  procure  an  aircraft  before  a  mission  is 
scrubbed.  As  shown  in  Figure  A-10,  when  the  aircraft  is  needed,  the 
DUTY  SCHEDULER  first  looks  for  aircraft  in  this  order:  in  the  FSC 
POOL,  in  the  OR  MAINTENANCE  POOL  not  actually  undergoing  maintenance, 
and  finally  in  the  OR  MANTENANCE  POOL  actually  being  worked  on.  When 
an  OR  aircraft  is  found,  maintenance  is  discontinued  immediately;  the 
aircraft  is  transferred  to  the  TA  WORKSHOP  for  flight  preparation. 

If  the  aircraft  is  NOR,  the  components  or  FM  requirements  causing  the 
aircraft  NOR  status  must  be  repaired  before  flight  is  possible. 
Facilitating  of  the  repair  can  be  accomplished  by  making  sure  that 
people  are  available.  Since  assignment  of  people  is  based  on  a 
priority  system,  raising  the  priority  and  reassigning  people  expedites 
the  job. 

This  priority  repair  can  occur  in  two  situations: 

1.  NOR  failure  is  discovered,  in  turnaround  inspection,  that  requires 
less  than  4  hours  to  repair  (based  on  parts,  facilities,  and 
anticipated  time  to  repair).  The  repair  is  accomplished  using 
personnel  from  a  special  work  center  on  the  flight  deck  (in  the  TA 
WORKSHOP) . 

)  2.  An  aircraft  is  required,  none  are  available  in  the  OR  MAINTENANCE 

POOL,  and  there  is  a  NOR  aircraft  that  can  be  repaired  in  less  than  4 
hours. 

In  both  of  these  cases,  the  aircraft  is  placed  in  the  PRIORITY 
MAINTENANCE  POOL  with  a  high  priority  exceeded  only  by  priorities  for 
turnaround  and  daily  inspections.  Priority  repair  is  not  permitted  on 
aircraft  with  overdue  PM  requirements. 
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